Home » testing

Tag: testing

How to test an idea

A mistake that I made more than a couple of times is to have a “bright idea” and to start implementing it right away. This approach is a problem because I had an audience of ONE. 

A much better approach (that I learned from Seth Godin) is to have a bright idea and send it to people in your network and try to get at least ten fans that will cheer you on as you develop this idea. You also need to delight these ten people so that they will tell the others. 

Finding the audience first is much harder initially, but its beauty is the marketing is built-in. You already have your sales team that will promote your product for free. Your idea will spread.

It is effortless to imagine that if I like something, then many other people will like it too. So if I build a product for me, others will line up to get it, correct? Not so fast! 

Indeed, others may like the same thing, but it will be tough to find them if they are not already in your network. When you build the product first, you get stuck trying to find people to sell it to. 

In the other approach, you build the audience first and then create a product they ask for. Not only will you have lots of feedback to make the product very relevant, but you will already have a waiting list of customers :). 

What if my idea gets stolen?

The quick answer is this quote from Tim O’Reilly: “Piracy is not your enemy… obscurity is.”

And idea by itself is not worth much. Without an audience and without you to execute it, the idea remains just a nice concept. 

The generous way to move forward is to share your ideas and then develop the ones that stick.

don't waste time creating new products.

Don’t waste your time creating new products

Here is a mistake that I have made way too many times. 

I have this bright idea for a new product or service. I am so excited about it! But, I want to keep it a secret least someone would steal it from me. 

I work hard to implement it, test it, polish it. I am making sure it looks like how I imagined it. 

After months of effort, I finally launch…

… to crickets. 

There is no one out there who cares about my product or service, let alone steal it. 

The problem is that this is a selfish way to create a product or a service. You are choosing to work in a void in your head, and you are not doing it for an audience, so you never bother to ask for feedback or even ask if they need such a product. 

A better way is to test your assumptions before you spend any time building stuff. Building a useless product costs you time and money that you could have used to make something relevant and remarkable.

Testing can be as simple as saying this: 

“Here is what I am working on next. What do you think? Would you spend $100 for early access and an opportunity to give feedback on how I make it?”

Two things can happen:

  • less than ten people signup for the early access: good! – you thank and refund everyone and let them know there is not enough interest to build the thing. Don’t skip the “thank you” part, as they are your biggest fans! And now you just saved time, money, and effort that you can put into testing the next idea.
  • a lot of people signup – good! – the pressure is now on to build the thing, and you have feedback from people that will help you make it very relevant. You also get testimonials for the official launch date. Again, make sure to reward the early adopters for making this possible.

Developers like us do things like this…

Developers like us use version control because we understand the value of being able to roll back.

Developers like us do backups because we understand our customer’s need for safety and insurance.

Developers like us have a process because that is key for delivering quality results consistently.

Developers like us ask questions because we understand the pitfalls that come with assumptions.

Developers like us understand business because the perfect solution, delivered too late, is no solution.

Developers like us are good communicators because we don’t expect the client to navigate technical jargon.

Developers like us are honest, simply because honesty is good for business.

Developers like us own our mistakes because it is the way to build trust and get hired again.

Developers like us don’t hide bad news because it shows care for the client to let them know you will not meet the deadline. It gives them time to plan accordingly.

Developers like us value privacy because the client needs to know their private data is safe in your hands.

Developers like us are generous because helping others along the way makes things better.

Developers like us are flexible because it is not always possible for the client to adapt to our workflow.

Developers like us are more expensive because we always deliver more than we got paid for.

Developers like us minimize risk because they understand the client has their reputation to consider when she places it in our hands when we deliver a solution.

Developers like us work fast because we don’t reinvent the wheel and use the best practices available in the field.

Developers like us never stop learning because we know first hand how fast the software world changes.

Developers like us future proof their code because it is never safe to assume how it will be used later on.

Developers like us prioritize customer needs because the final product is for them, not for us.

 

Note: this is a manifesto based on Seth Godin’s idea of “Tribes”: people like us do things like this.

Still not using Log Files in your app?

Have you ever had to contact support for a web app or a plugin to fix a problem, and the first thing they ask is for full access to your web server so they can “debug” the issue? 

This request frustrates me to no end. 

It is unprofessional, and it is lazy. 

The reason support asks for this is so they can run tests and inspect the results on your LIVE server. If that makes you nervous, it should! How can you know that they will not accidentally mess with your customers’ data? Not to mention all the privacy issues that crop up as soon as you hand your keys to a third party with no control. 

A proper way to deal with providing support for your app or your plugin is to add logs—a log file journals the activity and the data passing through your code. Inspecting a good log file will almost always let you know what the problem is and where the problem is. When a customer calls you for support, you only need to ask for the log files, not the keys to the server. 

In my experience, a good log file creates a breadcrumb trail that documents the data flow and the branching decisions in your code. Ideally, inspecting the log file alongside your code allows you to precisely follow along and determine what was wrong, without even having to run any code. 

A common mistake is to be unnecessarily verbose while at the same time not documenting the branching decisions. Silently discarded errors and exceptions are the usual pitfalls, and close second are if/else branches where only one of them leaves in a mark in the log. 

Security and Privacy

Now that you understand why log files are a must, especially in a client-server situation (like all the web applications), you need to be careful not to store sensitive data into the log file. Don’t store passwords or credit card numbers, and unless absolutely necessary, do not store emails. 

If sensitive data is required for you to be able to rebuild the data flow, make that available under a specific “log level” that is only activated on request. And in some cases, the entire log system can be activated only when trying to debug a problem. With this approach, however, you lose historical data that you need to fix the problem.

Always provide a way for an admin to flush the logs. 

Rolling Over

I am an overly enthusiastic user of log files. Simply because they work, and they speed up the process of solving problems. But there is a mistake that I kept doing for far too long. That mistake was no automatic rolling of the log files. What that meant is that the logs grew and grew until they would eat up all the allocated disk space. 

Oopsy! 

When using log files, decide when a log entry is too old and have an automated mechanism to remove those logs. Rolling the log files once a month (log1, log2, log3, etc.) and removing the very old files is a useful approach. 

If you don’t currently use log files, what is your strategy to support and debug your application while it is running on the customer’s LIVE server? I hope you will not say: “get root access and hack away until I find the bug” 🙂

Automated “downtime” alerts

Do you know that frustrating moment when you realize that your website has been offline for three days? Or that your shopping cart stopped working last week? 

That moment is also valuable because you now know that something is broken, so now you can fix it. But at the same time, you wish you learned of this faster!

On a community website, this may not be an issue, as your users will let you know when the site is broken, but that is not the case for a blog, or an online store, or a landing page that is collecting leads. 

You could set a daily reminder to check things are OK, but that will chip away at your precious time, and it quickly becomes boring, so you will begin to forget to do it or begin to think that you don’t have to monitor the website anymore. 

I am all about automation, so let’s automate this! 

Google Analytics

The easiest way that is also free is to use Custom Alerts from Google Analytics. The logic is simple. You have an expected value of daily traffic (based on historical data), so you create a custom alert to let you know if it drops below that. Of course, you need to have Google Analytics installed on your pages for this to work. 

Pingdom

Another way is to use a tool like Pingdom. I have used them for a long time in the past. They no longer have a free tier, but the value you get from the service I think is well worth the $10/month they ask for it. I like Pingdom because they provide more than just “your web site is down” notifications. They provide performance analytics too, which, as we know, is a factor in how your website ranks in Google searches. 

But the real power of Pingdom is transaction monitoring. Transaction monitoring helps you know if a process is working, not just a page: a process like the signup form, or progressing through making a purchase. These are incredibly difficult things to set up tests by yourself, and you get that for $10/mo.

In House Tools

You can also write mini scripts that load your webpages and inspect the results for clues to determine if the page functions as you intend to. Since I am a software developer, that is what I use today for most of my projects. 

The downside is that you have to write these scripts, test them, and maintain them. Depending on your team composition, that may cost you more than using something like Pingdom. 

The upside is that since it is your code, you can do all sorts of interesting things with it, not just email notifications. You can use that to trigger different processes and even attempt an “auto-fix” by restarting relevant processes or clearing out the caches. 

A more powerful subset of this is writing automated tests for your web apps using a tool like “TestCafe” to simulate a user interacting with your web application going through a purchase or signup process. 

You can create custom monitoring and analytics tools to aggregate data from multiple signal sources that can provide insights not readily available in Google Analytics. For example, you can monitor how a campaign is affecting not only your website but also social media engagement across all the networks you care to track. 

Event-Based Programming

After you work long enough on software projects, it will become self-evident why complexity is your enemy. Pieces of code that are highly dependent on each other will result in a maintenance nightmare. You cannot change or upgrade anything without risking to break the different parts that are tightly connected to it. 

The solution I have found that works best is “Event-Based Programming.” I did not invent it; it has been around for a long time. I discovered that adopting this pattern has made maintenance much more straightforward. 

In a nutshell, your program is no longer a collection of functions that call each other in an ever-increasing web of complexity. Instead, you have components that talk to each other by raising or listening to events. 

This breakdown allows you to change each event generator or event listener individually, and as long as the event format does not change, you don’t risk a break down in communication. 

An event generator will say: “Hey, something interesting has happened, and here are the details.” And it does not care what happens with that announcement. It could be that nobody cares, or it could be that many will take action on that event. 

An event listener, on the other hand, does not care how an event was generated. As long as something interesting happens, it will act on it. 

This decoupling makes debugging super easy too! Because you can test components independently by merely looking at the kind of “chatter” they generate. 

If you’re reluctant to adopt “events” in your codebase, now it’s time to make the jump.

Using WordPress as a Prototyping tool

Creating a prototype for your product or service is an excellent idea. It allows you to do some research before you commit to a specific solution. You can test various layouts, various interactions, and you also have something to show to your investors or your target audience to check with them if this is something they would spend money to buy. And the cheaper the prototyping, the more tests you can do, and the more information you will have when you want to build the real product or service. 

A prototype has only very basic functionality, and it is used to get an idea of what the user interaction will be like and what possible challenges may show up in future development. Because it is so simplistic, you can choose any technology you want to create your prototype. It does not have to be the same tech you will use for the real product. And this is important because you can choose something free, or something that you know how to operate.

In this article, we will focus on WordPress. Why? Because it is free, it is a common platform, it has a rich plugin environment, and the new block editor makes it ideal for quickly creating complex layouts. 

All you need to do to get a free WordPress site is to head to WordPress.com, create an account, and start a new website using their free plan, and now you are ready to begin prototyping. 

Go to your website’s dashboard and create new pages using the block editor. It is almost like using lego pieces to build something. 

You can create multiple layouts for the same test page; you can create links between pages, simulating a user interaction, you can test how it will look like on a mobile screen, play around with various font faces and sizes and so on. 

As a bonus, you can activate the comments feature and use that to document feedback on the pages you are putting together. 

Using the Free plan on WordPress has some significant drawbacks:

  •  you cannot use custom CSS to finetune your design
  •  you cannot install plugins 

Depending on the complexity of the prototype, you may not need either one of them, but if those are important to you, then you can install WordPress on a shared hosting plan and unlock the full power of the platform. 

Closing Notes

I know that WordPress is not a prototyping tool. For more advanced use-cases, you will find it limiting. In those situations, you are better off investing in some dedicated tools like WebFlow, or Sketch, or Figma.

But as a playground, when you want to get a feel for how your idea will look like, and you want to have something to show to your investors or your audience, give WordPress a try.

Finally, all prototypes should be discarded when you go build the real thing. There is a big temptation to use the prototype for the live product, but please don’t. When you need to make something that is production-ready, start from scratch and do it right.

You have a great idea? Make sure your audience thinks that too :)

Some time ago, when predictive text typing was a “thing” on mobile devices, I had this great idea of adding the exact same features on desktop computers too. Just type in the first few letters, and the computer will make suggestions to help you finish the word. You would type faster and make fewer mistakes. What a great product, right?

I have spent one year creating and polishing the product, building the website for it, and also spent some money to have magazines review the product. 

What was the net result of all this effort? 

It was exhilarating to build this thing and overcome the tech challenges, but in business terms, it was a disaster. I sold just about three copies of this software. 

There is a valuable lesson I learned from this: test your ideas first, before actually building the product and service.

The good news is, in today’s world, you can test your idea for free and with very short setup time. And this post, I will share some tools to get you started. 

Social media

Social media is the most accessible medium to use and the quickest to get feedback on. Write a post about your idea, and ask for comments. Do you get any engagement? Are there any enthusiastic fans? Does it make sense to start a group around this on Facebook? Before you know it, you have your cohort of fans that will support you in building the “real deal.” And you will also have some precious feedback about how to build your service to serve your audience best.

A free website

Before investing thousands of dollars for a pro website, test it out with a free service.

WordPress.com is a good tool to use. Start with their free plan. WordPress is the one I use the most for myself and my clients.

Wix is another platform that has a free plan. I haven’t used it myself, but I’ve seen it working. If what you need to test out requires you to show some sort of design and less text, then I would go with Wix because of their powerful design editor that is not available in WordPress.

WebFlow is new to the game. It has a free plan, and it is aimed more at designers who what to greate outstanding pages without having to write code. If you are a designer, you will love WebFlow over the other options. 

Some important tips to consider

Intellectual propery

If you do not want to test your idea because you are concerned, someone else might steal it consider these: maybe your idea is not different enough, and that makes it super easy for anyone else to replicate? In which case, you may be better off pursuing a more innovative idea. Also, even if someone “steals” your idea, they don’t have you on the team with your insight, thoughts, and experience. There is a long road from an idea to a mature project :). 

From testing to an actual service or product

Regardless of what platform you choose, think about how will you grow if the idea turns out to be viable? You need to be able to export the content you have created for your tests. And it could be images, articles, or the design itself. It is always a good idea to document what you are doing for two reasons: if it has not worked, you will be able to learn something from it. And if it does work, you will have built a library of resources to help you launch the mature website/product/service.

The Process – A project from start to finish

There is value in having a process. It helps you provide consistent results, and you have something that you can continuously work on to improve. 

Here is “The Process” that I use today with software projects.

1. Are we a good fit? 

The first thing that happens is the discussion where both myself and the client try to determine if we are a good fit for one another. 

My job is to determine what the client needs and consider if I can provide a creative solution to solve that problem at a price that is fair for both of us. 

Sometimes this discussion happens in two parts if I need a break to do some research and investigation before I can begin to think of ways I could help. 

An important note here is that what is “needed” may not be what the client initial thought may be needed. That is why we have a conversation before any agreement happens. 

2. Project Set-Up 

After we agree on scope, price, and what it means to be 100% satisfied, I begin the work. 

With time I have learned the value of keeping things organized and tidy. 

Each client gets their individual folder that will document the history of the project. In that folder I will have things like:

  • meeting notes; 
  • agreement of project scope and price;
  • backups – I never do any changes unless I have a backup first;
  • client files – images, documents, other media;
  • a work-log – where I document what I have done and why. In very rare cases, I can use that to remind the client of the road we took together and justify a decision over the other. Another benefit is that you learn and get better by journaling what you do; 
  • access details – a file where I store various logins that the client has shared with me. In some cases, it makes good sense to have this file encrypted, like a ZIP archive with a password, for example;
  • any new agreed-upon changes also go here;

3. Making a plan – The List

I was trained, mostly by my father, to be organized by using lists. And I have kept that training and added on top of it. It is very useful, and it gives me clarity on what it needs to be done and in what order. 

Here I make a list of everything that needs to be done, broken down in tasks. The tool I use most of the time is Asana. I have tried Trello and Bootcamp, but I find Asana to be much closer to how I like to work. 

I also use a calendar service (like Google Calendar) to remind myself of upcoming deadlines.

Something that I found is handy is to split my list into three main sections:

 a. Go Live – the project cannot go live or shipped if any of the tasks here are not finished;

 b. Nice to Have – other tasks originating from the client that we can add later, after the go-live and in, some cases, in a “Phase 2” of the project;

 c. Bright Ideas – here I write down my own ideas that I think could help the client;

Why am I organizing things like this? 

The short answer is that it forces me to focus on the client; to get them on the market as soon as possible. I did not always use to think like that, and I was routinely making the mistake of focusing on tasks in the “Bright Ideas” section because there were so inspiring to me and they would challenge me. But in most cases, they were not mandatory for the client. So that added unjustified delays and extra costs. While I don’t think the client is always right, I do believe the client knows what is important to them. And that is where my focus should be and what I should be solving first. 

I hope it is now evident that the order in which I go about these tasks is: Go Live, then Nice To Have and then Bright Ideas. And I have learned to be OK with the fact that most projects stop after the “Go Live” part when the burning need of the client has been met. 

So why still keep “Nice to have” and “Bright Ideas” around? 

The biggest reason is to clear your mind so you can focus on the tasks at hand, knowing that your “good thoughts” are not lost. The second big reason is that is how you learn and grow. Maybe you don’t implement these ideas now, in this project, but because you wrote them down and thought about them you will remember them, and they may be the perfect solution for the next project or the “Phase 2” of this project. The “Bright Ideas” section is your most creative section. Don’t throw it away.

4. Set Up a Schedule 

I believe that if a project does not have a deadline, it will never finish. I am very wary of clients who say: “we can finish this whenever… no rush!”. That can be a source of significant delays for you and the project. 

I know this is not the same for everyone, but if deadlines motivate you, a client who is continuously delaying the project will drive you mad. 

In this step, I set-up reminders in my calendar for milestones that will help me get the work done in time for launch. 

When I do this, I need to allow time for the “unexpected” right before the launch. So I plan to finish the project at least a few days early to have some space to extend in case of unforeseen trouble. 

5. Doing the work

Only in step 5 comes the most fun part for me, doing software work :). But as a solopreneur, I need to do and master the business admin part as well.

When doing software work, I have a few principles that I follow:

  • Blocks of uninterrupted time – 2 – 3 hours blocks when I am the most efficient. When coding, there is a complex context that you need to have running in your head, and that takes time to build. If your block of time is too short, then most of that is spent just reminding yourself what the project is about;
  • use a versioning system – this should be obvious – even if you are working alone, it is so much easier to roll back to something that was working when you have a versioning system in place 
  • automatic testing – for particular clients that require a very high level of quality control this needs to be done;
  • Automatic backups of the client’s old code/website – again, just in case you need to roll back. Make sure the backups also include the database, not just the code files;
  • Keep track of working time – in some cases, “hours of time” is what I am billing to the client, and also this is how I know if we are going to finish on time. I am however transitioning out of this, so stay tuned for a post about it. Keeping track of time can be a learning and discipline building tool, just like journaling your work, but sometimes it becomes very, very restrictive and creates a lot of stress; 

6. Client Feedback 

I used to work, work, work, and then do the “big WOW” reveal at the end when the client would be floored with the amazing quality and results. 

This big reveal was silly. 

Why? Because it would allow me to focus on the “Bright Ideas” list instead of “Go Live.” And I would deliver an excellent, high-quality product, that would not speak to the client’s needs. 

What I do now is to deliver work in smaller increments and get feedback soon and often. I am careful here that the feedback I am looking for is “does this meet your needs, madam client?” and not about “how to do my job.” Therefore it is a tool to keep me focused on finding solutions that are important and relevant to the client. 

7. Making mistakes 

Mistakes are happening in every project. If I am not making mistakes, I am not learning anything new, and I am just delivering the same old solution. In some cases, that is OK, but generally, that is not what I am looking for. Each client is unique, so I want to challenge myself and meet their individual needs. 

It is therefore essential to know that I will be making mistakes and have a plan on what do to about them. Like, make sure I have factored that in the price so I don’t add more as a cost to the client to fix them. Also, I need to include those in my schedule. That is why I always have the “unexpected problem” in my planing with some time allocated to it. 

I am, however, fair. If I make a stupid mistake that I could have easily foreseen and avoided, the fix is on the house! I am talking here about the unavoidable trials and errors when you are building custom solutions and exploring places you have not explored before. Those mistakes need to be allowed for if you want to arrive at a good solution where you have explored alternatives that did not work. 

8. Going Live 

When I “go live,” I strive to have a seamless experience for the customer and their clients, which means as little to no downtime if that is at all possible. Over time I have discovered various ways of “flipping the switch” that can use depending on the specific situation. 

Again here, backups are super important. In case you mess up the live deployment, you need to be sure you can roll back to what worked before. I have seen so many instances where this not done, and people roll the dice. They may be that good, but it is just a matter of “when” things will crash on you, not “if.” I have learned my lesson.

9. Review and Learn

Ideally, the project has completed with the “double thank you.” You give thanks for the business, and the client is grateful for the solution they got. 

Either way now is the time to reflect on the project and look for “lessons learned.” 

What worked? Do more of that!

What did not work? Do less of that! 🙂 Or at least try to figure out what you can do differently next time. 

Something else that shows up here is opportunities to learn new things. Look at the “Bright Ideas” section and the “Nice to Have” section and try to come up with solutions to those that use new technologies. I sometimes set-up a pet project where I implement that solution.

I mustn’t skip this step or else I would get stuck in a rut. 

Mic to you!

How is your process different when you’re helping your clients? What are some of your lessons learned along your journey? 

Keep your site up to date – good advice, but only in theory?

“Keep your software up to date!”

I am sure you have heard this saying many times, and in general, it is good advice.

However, let’s take a WordPress website, for example.

A WordPress site is made from the WordPress core and usually many plugins. You have many pieces in your puzzle. Not all of them are updated at the same time or in the same way.

In my experience, it happened more than once that an updated piece no longer fit with the rest of the puzzle. Auch!

Most update processes do a good job warning you that you need to do a backup first and to ensure the other plugins (components of the puzzle) are compatible with the update.

This approach puts the responsibility of “making sure that things still work” with the user. And not everyone can make that assessment. Also, let’s admit it, sometimes we are in a hurry or just plain lazy :). Ideally, a piece of software should not rely on a human to do the right thing.

I too used to be overconfident in the automatic updates process, and I would apply those every time there was a new update. Click, click, and I was done! What could possibly go wrong?

One time, I updated the store. We had a newsletter scheduled that we expected to generate much interest and I thought I wanted to offer the best and latest shopping experience for our customers. So I updated the store and was on my way. The next day I opened the email to a ton of complaints from our subscribers that the checkout is not working! A full email campaign wasted, not the mention that we looked totally unprofessional — that hurt both my ego and the sales.

What did I learn from it?

1. Updates can potentially be very painful

2. You never change the system right before a big promo campaign. (This feels like common sense now.)

3. You need to be extra careful when you update the part that generates income: the store, the “pay now buttons,” and the subscribe boxes.

4. After you update you need to test at least the critical functionality: add to cart, checkout, subscribe.

5. You better have good backups, in case you need to roll back.

It was not all bad, because I did have backups, so it was relatively easy to go back to the previous version. Moreover, I sent another email campaign with an apology and a second invitation to check out the offer. If I hadn’t had backups that would not have been possible to do.

Another thing I decided to implement is to write automatic tests for the website that can run in the background and make sure the critical processes are still working.

In conclusion: stay up to date, but be smart about it 🙂