How to create quality programming challenges

Programming challenges are a great way to teach and assess the programming skills of your students or potential hires. In a professional environment, they help to filter and recruit the best software developers. Coding challenges are also crucial in teaching programming to future developers, no matter their age. The satisfaction and rush they get from solving a programming puzzle is a big motivational kick into pursuing a career as a developer.

Thinking about running a competitive programming event or a hackathon? These types of events often revolve around solving coding puzzles quickly and efficiently.

In this post, we will go through the basic steps and common pitfalls of creating and testing your own programming challenge, which you can later on use inside your hiring platform or as a part of your curriculum.

1. Get inspired

At the earliest stage of the process, look for an interesting idea that will become the foundation of your programming problem.

It may be a real-world issue that you always thought could be improved. The more passionate you are about the topic, the bigger chances are that your programming problem will be interesting and engaging for the problem solver, however, remember not to lose touch with reality (not everyone is as passionate about the details of string theory as you are ;).

2. Design the challenge

Once you have an idea, you need to forge it into the problem that you want the programmers to solve.

That’s a lot of conceptual work to do. Grab a pen and a piece of paper to dig into the essence of the problem, draw diagrams and figures, look for edge cases, and consider the most promising solutions to the problem.

This may sound like a struggle, but don’t feel intimidated and be willing to look for relevant help from fellow programmers, lecturers, the developer community, or simply Google. Computer science books might also come in handy at this stage - "Introduction to Algorithms" is always a good starting point.

Don’t forget to set the problem to the desired level of difficulty. A puzzle that is too hard can discourage entry-level developers from solving it. On the other hand, an overly simple challenge won’t be exciting for more advanced problem solvers.

Remember that this is a very creative process and takes a mix of imagination, math/computer science skills, and luck.

3. Create a formal description of your problem

Now that you have the algorithmic problem and solid knowledge about its properties, you need to write down its formal description - the essence of the challenge expressed in the mathematical language.

Here’s an example of how a formal description might look like:

Let K be a natural number, not bigger than 30. Sort an a_k sequence of K natural numbers (where 30 < a_k < 250 for every k in the range from 1 to 30) in ascending order.

Sounds scary? Don’t worry - it’s only for internal purposes. You’ll use the formal description as a reference - it’s very useful during the story writing and testing processes.

4. Turn your challenge into a story

At this stage, you still have nothing exciting to show. The formal description is useful, but (to be honest) it’s dry and boring. You want programmers to be really excited, so you need to turn the problem into an engaging story.

This is one of the most challenging and tricky parts of creating the challenge as it requires some storytelling skills. You want to make the problem engaging, simple, and precise.

From the technical standpoint, this story is the problem description and it’s the single point of reference for the problem solver. It must be as precise as possible - if some details are important to solve the challenge, don’t allow them to stay in your head and put them inside the problem description.

You can find many great examples of problem descriptions on our Spoj community website - start with the "Food Shortage in Byteland" problem.

5. Adapt the challenge to an online judge

As soon as the problem is well defined and you have your story in place, you’ll need to adjust the challenge to the scheme of an online judge - the system evaluating solutions submitted to a challenge by the problem solvers.

This is a crucial stage, at which you need to carefully and thoroughly do your technical homework and:

• implement the model solution (sometimes many of them) to prove that it’s possible for the developers to solve the problem,
• prepare test cases - data sets that the online judge system uses to test the submitted solutions,
• adjust the scoring system or, for the most advanced challenges, implement it from scratch.

Even with a very good project, creating a challenge may be difficult - consider using our Problem Setter’s Handbook for in-depth technical knowledge on how to create quality programming problems. This handbook accumulates years of academic experience and points to the best practices when it comes to creating programming challenges.

You can also sign-up for a Sphere Engine account and use our built-in editor that will guide you by the hand in creating your first programming problem. After you finish creating your first coding problem, you can quickly put it online or export it to use in a manner of your choice.

6. Test the challenge internally

When you reach the point when the challenge is complete, there is still work to be done. Verify each piece of the programming problem during the internal testing phase.

Confirm that the formal description is consistent with the storyline. For the model solution, you need to make sure that it’s correct and compatible with test cases.

Be humble - tests done on your own are not as good as those carried out by someone who hasn’t been directly involved in creating the challenge.

7. Test the challenge externally

At this stage, you have a growing feeling that the programming problem is complete and ready to go. What if you’re wrong? Reality shows that even the most talented people need an extra pair of eyes to revise their work.

Look for a helping hand amongst your colleagues or coding communities you’re part of. Reach out to testers who will be able to:

• understand the challenge,
• solve it,
• confirm that it represents the desired difficulty level and meets all other requirements,
• suggest potential changes and adjustments.

Give your testers precise guidelines regarding what they should test and remember that two testers are better than one, three testers are better than two, and so on...

For an in-depth look at creating programming challenges refer to our Problem Setter’s Handbook.

Got any feedback regarding this guide or looking for help in creating challenges? Contact us!

At Sphere Engine we combine years of experience in recruiting developers and assessing coding skills with the latest technologies. Our APIs and software components enable creating programming challenges, online compilers, IDEs, and teaching coding to millions of end-users across the globe. We work with companies like Infosys, Amazon, Edhesive, support thousands of educational institutions around the world, and run our own community-focused platforms - Spoj and ideone.