Design Challenges: Designers Demand Reform as Unfair Tasks Miss the Mark (part II)
Examining the ethical, psychological, and practical concerns surrounding design challenges, and advocating for a more inclusive, human centered and effective approach to evaluating designers.
Table of contents (part I) To read the first part of the article click here
1. Background context
2. Methodology
3. Findings
4. Objective and/or Scientific Foundations of Designers Complaints
5. Negative Impacts for Businesses
6. What designers wants?
Table of contents (part II) click here for the second part
7. What motivates designers to promote design challenges?
8. Why do designers who advocate for design challenges continue to insist?
9. But what about whiteboard challenges?
10. I’m still going to ask for a design challenge
11. Conclusions
What motivates designers to promote design challenges?
Over the years, colleagues who promote design challenges, as well as whiteboard challenges, have given various reasons, and, personal perception, have changed slightly over time.
It used to be common to hear “because I need to evaluate how you work under pressure” but at least in Europe, you hear less of that and more of “I need to see how you think” or, more recently, almost as if feeling cornered, “how else can I evaluate you?”.
In reality, none of the reasons that promote this practice have any scientific basis, in the sense that no one has ever bothered to conduct a study to determine whether speculative work is a valid system for evaluating a candidate.
Below are some of the most common reasons designers says they ask their colleagues to conduct a challenge:
— Demonstrates a designer’s problem-solving skills
While it’s true that design challenges could provide a snapshot of a candidate’s problem-solving abilities, the limited scope and artificial environment of these challenges may not fully capture a designer’s ability to tackle complex, real-world problems.
As highlighted in this article, design work is multifaceted and often requires considering various aspects.
A design challenge focused on a specific task or scenario may oversimplify the design process and fail to assess the broader skills required by the profession.
It’s worth to mention, that often design challenges do not present a problem, they present you the solution:
“Design an app that does this and that”
and on top of this, a portfolio typically already showcase the designer problem-solving skills.
— A glimpse into a candidate’s design process
Design challenges can indeed give some insight into a candidate’s design process.
However, without adequate feedback or dialogue, candidates may not have the opportunity to fully explain their thought process or the rationale behind their design decisions.
Furthermore, the time constraints and high-pressure nature of design challenges may not accurately reflect a candidate’s typical design process.
— Offers an opportunity for designers to showcase their skills
This is really the most classical inherent motivation of a speculative work but in reality a portfolio already showcase skills.
— I have 1000 candidates I cannot go through all their portfolios
I cannot verify the validity of this comment, perhaps it is so in the United States or perhaps is true for large businesses or very desired one, for example I kinda remember that NNGroup gets around that amount or more when they open a position.
If I have to give an evaluation based on personal experiences, here in Europe we are full of small and medium-sized companies and when during the first call I ask among how many I have been selected, the answers generally always revolve around these numbers:
“100–80–50 applications” and 10 selections of which then about half is the one that passes to the design challenge stage.
— I receive many applications from candidates who have done a bootcamp and their portfolios are full of not real projects
Just because someone’s portfolio contains projects completed during a bootcamp doesn’t diminish the quality or relevance of their work.
Students of the four main online bootcamps actually interview real people for their course projects, thus making these portfolios projects more realistic than the fake prompt design challenges without dummy content that designer evaluators assign to the candidates.
Motivations like “because I need to evaluate how you work under pressure”, “I need to see how you think” and “how I evaluate you” have all been indirectly addressed above in this article.
— A designer it’s a cost, we need to de-risk the chances to pick the wrong designers
I don’t have research data that could specifically object to this comment, I could explain it on a psychological level, but I think it’s not necessary considering what has already been written so far.
Designers could easily argue that all employees have a cost.
Depending on the company, country and city, some employees cost more than a designer and yet they don’t have to do a design challenge with no data and which requires hours of work.
Don’t make the mistake of thinking that in every country, even in Europe, a UX Designer or a Developer is always paid more than, for example, a Recruiter, a Marketing Manager or a Customer Support Team Leader, because this is not always the case. That is perhaps more of an American, German and some Northern European countries reality.
Some designer (and they do) might say that doctors are not asked to perform a test surgery on a dead body, nurses are not asked to sew an orange to see if they know how to stitch, paramedics are not asked to drive the ambulance from the point A to point B as quickly as possible and pharmacists are not asked to guess the correct drug based on the person’s symptoms or make a preparation in the laboratory.
At least, on a personal level, I’m sure of it because I come from a family of doctors.
— Yes but doctors have a certificte, a degree etc
Architects and Graphic Designers too, yet before internet they were among the first ones to experiment speculative work.
Plumbers may have a diploma, yet when they enter into a firm, I doubt they are asked to perform work for free as part of the hiring process.
Why do designers who advocate for design challenges continue to insist?
Let’s set aside strictly personal reasons like:
- Someone might seek a candidate primarily engaged in crafting without the need of knowing UX Design, an know-how by osmosis may be considered ok
- Business reasons (eg I’m the CEO I decide)
Despite detractors of this practice having solid and valid grounds to oppose it, in a field like UX Design (when practiced correctly), where decision revolves around data research, why do proponents of design challenges or whiteboard challenges persist with this practice?
Personally, I think that the motivation are simply psychological and social.
For example, designers has already clarified that they do not want to work for free and that a portfolio walkthrough and a peer to peer interview are they way to go, so why a design challenge promoter should reply with:
“What is the alternative?” when designers has already presented the alternative.
Perhaps D. Norman would reply by saying that this is the symptom, not the problem.
At this point you should apply critical thinking to the data that now you know and a bit of metacognition and then ask yourself key questions like:
- Why do I still feel the need of an alternative different by the one manifested by my colleagues, some with 5, 10, 15, 20+ years of experience?
- Why other businesses do not ask for a design challenge and still their design teams succeed?
- Are my motivations backed by data?
- Am I victim of a bias?
- etc etc
Psychological and Social Motivations Behind the Design Challenge Promotion
If you ask me, as I previously said, there are probably psychological factors that are playing here, for example:
— Status Quo Bias
This is a tendency for people to prefer things to stay the same unless there is a compelling reason to change.
Someone may see a design challenge as a standard practice in the industry, regardless of its validity, and so hesitate to deviate from this norm without clear evidence of a better alternative.
But even with a clear evidence and better alternative, designers don’t see them because they may end to experience resistance to change.
— Resistance to change
I talk about it in these article, you can skip the top part and scroll down till when I talk about the emotional cycle of the psychiatrists Kubler-Ross.
Imagine your child has a favorite toy that he/she play with every day. One day, you tell her/him that you are going to replace it with a new toy.
Your child might feel upset or worried because she/he loves her/his old toy and he/she don’t know if they will like the new one. This feeling is what is called “resistance to change”.
It’s when we feel nervous or scared about something new because we are comfortable with the way things are. Just like your child might miss the old toy, adults can feel the same way when something changes in their life.
Eh, you think it’s easy, but it’s not, because this psychological phenomen can be a very bumpy ride when we have a strong ego or when we have spent years with the same convinctions and built our personality and who we are on it, something that does not happen consciously.
RTC can causes in people experiencing an emotional anxiety caused by the prospect of a transformation or change that is taking place and this trigger another amazing thing called cognitive dissonance.
— Cognitive dissonance
I also talk about it in the same article I mentioned above. Here is what it is:
This can then trigger a series of other interesting circumstances such as defensive attitude, difficulty accepting change, confirmation bias, fear of the unknow
— Defensive Attitude
Some designers may adopt a defensive attitude when their practices are questioned or criticized. Asking for alternative way of evaluation, could be a way to deflect criticism and maintain the status quo without seriously considering the feedback provided.
— Confirmation Bias
Designers may be biased towards their existing beliefs or practices and may overlook or dismiss alternative suggestions that challenge their current approach.
They may only consider alternatives that align with their preconceived notions of effective hiring practices.
— Fear of the Unknown
There may be uncertainty or skepticism about the effectiveness or reliability of alternative assessment methods, particularly if they deviate from familiar practices. Designers may be hesitant to adopt new methods without clear evidence of their efficacy.
Ultimately we can also try to call in Swidler, A. Culture in action: symbols and strategies (1986) and quoting:
culture influences action not by providing the ultimate values toward which action is oriented but by shaping a repertoire or ‘tool-kit’ of habits, skills, and styles.
In the field of design hiring practices, the “culture” could be the established norms and practices within the industry or a particular company. This includes the use of design and whiteboard challenges in the hiring process.
Despite the lack of empirical data supporting the effectiveness of design challenges, and the numerous criticisms of this practice, it continues to be used.
This could be because it is part of the “tool-kit” that designers involved in the hiring process have inherited from their predecessors.
It’s a familiar practice, something they themselves might have gone through when they were job candidates. As such, it becomes a habit, a known method of assessing candidates, and is perpetuated in the hiring process.
The use of design challenges could also be seen as a “style” of hiring — a way for companies to present themselves as rigorous and thorough in their hiring practices.
Even if the effectiveness of design challenges is questionable, the very act of having a design challenge could be seen as a signal of a company’s commitment to hiring the best talent.
Why focused on crafting?
I have to take the long journey, so following for some paragraph.
Another motivations to why some designer still push on design challenges, could be due to a lack of understanding of UX Design and its poor penetration as an official job, from the physical market, into the digital market, which was already occupied by other professions before NNGroup went online.
From 1999 NNGroup started to promote the usability weeks and the user experiences weeks.
These were 1 and 2 days seminaries (perhaps there was a 3rd not sure) that were mainly talking about usability on different aspects of a website and it tooks a couple of years before seeing on their website the combined words “UX Design”.
Indeed first they were using the words “user experience”, then “UED” and then “UX Design” and I always wondered if Nielsen and Norman implemented “Design” in place of “Architect” to follow a market trend.
Bottomline, things from the USA to the rest of the world didn’t spread fast and neither in an homogeneous manner.
One of the leading usability websites in Italy, active I think already in 1999, wrote an article in 2011 about how people didn’t understand what UX Design was, suggesting that perhaps was because it lacked the scientific basis that Usability Engineering had:
Translated means:
The feeling is that User Experience is a happy label, but that it is not clear even to all its practitioners what it is. Strengthening the sensation is the consideration that it is not a discipline with (missing word) brought by scientific research (unlike usability engineering, which boasts lines of research in Human-Computer Interaction), nor with ad hoc design practices ( as is true for information architecture, which has an arsenal of choice in card sorting, content inventory, construction of controlled dictionaries and more).
By coincidence, in a recent article, Jakob Nielsen reminds us that this field was originally called “usability engineering”.
This was in Italy (at that time 4th richest country in EU), site of reference to promote usability, most important books from D. Norman were already published, perhaps translated in Italian and we are at the end of 2011, precisely 2 years before the first bootcamp launched a course in UX Design that was not instructed by a Usability Specialist, a UI Architect a UX Architect or someone with a degree in Psychologist, but graduated in Art and Graphics.
This shows us that things don’t always evolve as we expect and in spite several indications that UX Design was not an upskilling but an interdisciplinary job profession the scientific disciplines we saw in the previous part of this article.
If you remember Don Norman’s books and his words, he has repeatedly emphasized the multidisciplinarity of this profession, including its scientific basis and well, requirement to apply them.
When I was a teenager in Italy, around 1997, I used to design interfaces for browser games and sell them under the table. I naturally called myself a UI Designer because I didn’t design websites and these were UIs, like the ones we have once we log into a SaaS with the side menu and the data at the center of the screen.
I didn’t design websites for banks, e-commerce, etc. Back then, professions like Web Designer, UI Designer, Visual Designer, and Graphic Designer already existed.
When I started working as a Web Designer around 2000–2001, I worked exactly like everyone else in that time, with a “project mindset,” so there was no upfront research, I didn’t design anything based on data except the data coming from my common sense, what I had absorbed from other sites and what the client wanted.
We would do sketches, slice out the home page from Photoshop, create user flows, and make changes based on user feedback once online (if we were working with a regular client).
Otherwise, it would happen just like in the design challenge prompts that everyone hates.
A client would come with a list of what they wanted or would tell you to make a website that does this or that. You would execute the order, change something according to the client and that’s all.
In some cases, especially with graphic designers, they would ask you for a free draft of the project and then decide after choosing from other potential designers.
Furthermore, jobs like Web Designer, UI Designer and Visual Designer in USA were already into the online digital market before NNGroup lauch and between 2000 and 2006 we have traces of job titles such as “UI/UX Design” that many today blame but that in that time, was just the UI Designer job title evolving to accomodate this new thing called UX, regardless the fact that UX artifacts typically comes before wireframes.
The fact that most job postings in Germany (I lived here), Spain (did interviews there), Italy (I lived here), and Greece (I’m living here) are all oriented towards crafting, makes me think that UX Design has perhaps penetrated as a general upskill, Nielsen’s heuristics have become “common sense”, and the whole scientific part has never been received if not as example of good design founded on such scientific subjects.
This could explain why design challenges continue to be crafted without dummy content.
Here are some additional points to consider:
- The lack of scientific understanding of UX Design can lead to a focus on aesthetics and superficial features over user needs and usability.
Typically these design challenge prompt lack of user needs, user goals, pain points, the data/dummy content UX Design need to perform the job. - The emphasis on crafting and visual design can lead to a neglect of the research and problem-solving aspects of UX Design.
Often these challenges do not give time for user research (but many would not spent time for free on that) nor give you fake user interview answers to analyze and, the prompt it’s not a problem, it’s already the solution. - The use of design challenges can be a way to assess a candidate’s visual design skills, but it does not necessarily measure their ability to conduct research, understand user needs, and design effective solutions.
After all, nobody really cared about this things in Web Design.
But what about whiteboard challenges?
Even though I have collected about 45 comments specifically about the whiteboard challenges, I have not included them in the analysis. This is because I originally intended to write a separate article on whiteboard challenges where I was involving interviews with different Psychologists.
However, the vast majority of the reasons that designers have raised against design challenges also apply to whiteboard challenges which are not inclusive, present the same problems of lack of dummy content and they also have a version where the candidate it’s not informed in advance about what will be the context.
I’m Still going to Ask for a Design Challenge
If in spite of this 2 parts article, included the potential consequences for you, your design team and the business you are working for, you still want to implement a (paid is honest) design challenge then…
Try to implement it in a fair way
1.If the candidate has a portfolio, explain him/her why the portfolio it’s not enough, what you didn’t find in it, at least it’s a feedback in order to improve the portfolio and a respectful thing to do.
2. Offer the candidate these two options, better with the third one:
- Design challenge.
- Whiteboard challenge.
- Audit of a chosen site.
3. Craft an ethical design/whiteboard challenge
- Do not ask to redesign something of your product.
- Do not ask to design something of a product that align with your product (eg you work for an eCommerce and you ask to design a cart page).
- Do not ask for the source file.
4. Craft a realistic design/whiteboard challenge
- Provide dummy content (eg user needs, user goals, paint point, user flows, the answers of the user interviews, empathy map etc).
- For whiteboard challenges, provide the context in advance, before the actual interview/call.
- If you need, use the AI to craft user interview answers, paint points etc (I mean fake for fake, as you just care about the process).
- Do not offer the solution, let the candidate analyze the dummy content and come out with the solution that he/she thinks.
- Provide the solution “Design an app that does this and this and that” only if you work in a feature team, an agency on project mindset, or you do not care about UX because you need a bricklayer.
5. Clearly specify what are the deliverables
- Sketches
- Wireframe mid, high fidelity
- User flows
- Canvas maps
- Empathy map
- User Journey
- Customer journey
- etc
All these things require time, if they are not listed, the candidate is free to do not provide them to you.
6. Specify what are the criteria of evaluation
- Write them in a clear way.
- Do not let ambiguity of judgment.
- Craft the instruction to avoid a final “assumption VS assumption” but you wanna win because you have the knife in your hand.
One of the best design challenges I saw was coming from an international business, an LMS, that was clearly stating “we will not judge your assumptions”.
They were not stupid, they knew that even if they had provided some dummy content, it was still too open to various assumptions all valid without a validation.
7. The candidate is the end user, when you craft the challenge do it using plain language, use bullet points, manage white spaces and avoid lexical ambiguity.
(Do not bother to my IA, I am bound to medium freepaid options (pff)
8. Give enough time which means at least 7 days, but expect that some candidate may have even less than 8 hours to dedicate them.
Best ones are open ended.
9. Provide feedback after you have evaluated the challenge, if you do it via email, avoid templates as it’s not respectful. You are all paid handsomely, to hand write a response.
10. Remember to indicate in the job advertise, that if the portfolio is lacking of what you need (state what you need) a design challenge will be required.
11. Describe also how many steps the interview last.
12. Write the salary or the salary range in the job ad, if you are in Europe is mandatory by law.
The new rules will make it compulsory for employers to inform job seekers about the starting salary or pay range of advertised positions, whether in the vacancy notice or ahead of the interview.
Employers will also be prevented from asking candidates about their pay history.
Once in the role, workers will be entitled to ask their employers for information about:
average pay levels, broken down by sex, for categories of employees doing the same work or work of equal value
the criteria used to determine pay and career progression, which must be objective and gender neutral
Once hired, the employee is now entitled to ask what is the salary of the other employees performing the same job and you must provide the data.
Some example of bad design challenges
This was an unethical one required by a founder of a stealth startup, once I read it and I also saw that his neo UX Consultant had recently changed title from Art Director just to look as UX Consultat and didn’t have any experience in UX, I’ve dropped out from the process.
This other one was also unethic and asked by a Director of a famous automotive agency in Germany
This other also unethic and with several flaw addressed in this article, was proposed by a Head of UX which is also a point of reference for an important online platform in the design field
Ultima Ratio
Perhaps designers promoting the design challenge, can still change approach in another way.
When designers are interviewing other designers, they are simply conducting user interviews.
If you know how to conduct user interviews, that you can conduct one with your colleague candidates.
You can keep the portfolio walkthrough and integrate the step with a set of questions, or just use questions.
Prepare the research plan like you would normally do and decide between structured and unstructured interviews and for each question assign a value to the answer.
Obviously this will present all of a series of problems like biases that you need to learn to tackle, how to get common consens between interviewer about the value between each number etc, but at least you may produce data for an analysis of this evaluation.
Evan Sunwall from NNGroup is promoting a structured interview here at this page.
However, while I have some questions about it, I do not see it as an easy approach for designers who are mainly involved in crafting and lack the necessary selfanalysis to conduce unbiased research, beside the fact it may take times for them. Yet it could be a last resort.
Conclusions
While this article is not exhaustive, research process was heavily synthetized and does not address in detail specific situations such as a candidate who only needs to work on the design system of a product or simply produce the UI, it still provide several points of think about it.
The current state of design challenges in the hiring process raises critical questions about fairness, effectiveness, and respect for design expertise.
While they may offer some insights into specific technical skills, the limitations exposed throughout this article are undeniable. Unrealistic expectations, ethical concerns about unpaid work, and a lack of alignment with real-world design work, all call for a fundamental shift in how we assess design talent.
The lanscape is made by a complex interplay of historical legacies, cultural influences, and individual motivations. Despite the evolution of UX design from its scientific roots, there remains a significant gap in understanding and implementing scientific principles within the industry.
This gap manifests in hiring practices that prioritize crafting over human-centric problem-solving, perpetuating traditional assessment methods that fail to capture the essence of UX Design as a working profession rather than a field from which pick something.
The persistence of design challenges can be attributed to a myriad of factors, including status quo bias, resistance to change, and cognitive dissonance among industry professionals.
These psychological barriers, compounded by a lack of scientific understanding and standardized evaluation criteria, contribute to the perpetuation of outdated practices that undermine the profession’s integrity.
However, amidst these challenges lies an opportunity for transformation.
By embracing ethical considerations, promoting inclusivity, and adopting alternative assessment methods such a portfolio walkthough, collaborative talking, the UX design community can chart a path towards a more equitable and professional hiring process.
The conclusions drawn from this article underscore the need for collective action and introspection within the UX design community.
Challenging entrenched norms, advocating for change, and cultivating a deeper understanding of the profession’s core principles, designers can enter in a new stage of professionalism, integrity, and inclusivity in UX design hiring practices.
For further investigation, it might be interesting to explore through a more accurate and by the book research the whole topic and:
- The perspectives of hiring managers and why they choose to use design tasks
- The effectiveness of design tasks in predicting job performance
- The impact of these hiring practices on diversity and inclusion
