Submitting reports is one of the most important parts of any uTester’s job. This course will allow you to not only hone your skills in classifying and creating bug reports, but also how to take advantage of features in the uTest Platform.
How to file a bug report:
As you can see from the screen shot, click on the Report Issues button to begin to file a bug report.
Bug Report Template:
How to Classify a Bug
Testers must initially classify each reported bug with two fields: Type and Severity. Neither have any impact on payout. However, customers reserve the right to reclassify bugs as necessary and will assign a bug value which will determine any additional payout on the bug report. We ask both customers and testers to be fair with their decisions of classification and reclassification of all bugs.
- Graphical User Interface (GUI): Bugs that affect the presentation, look or layout of pages, images and form elements.
Examples: Misalignment, broken images, wrong sized graphic elements, inconsistent colors on a link, button or menu, etc.
- Functional: Bugs that produce an unexpected/illogical application behavior where the end result differs from the expected result.
Examples: Search returns the wrong results, clicking on a link takes you to page X instead of the intended page Y, etc.
- Technical: Bugs that produce error messages or malfunctions of the application.
- Critical: Bugs that are mission critical to the core functionality of the application and for which there are no workarounds. These bugs absolutely must be fixed before the customer can release the app to the public. Note: A critical bug is extremely rare and should only be used in instances where, if you were the product manager, you would delay the launch of a product due to this one bug.
Examples: Cannot log in, cannot check out or save my shopping cart, an email client that has a bug that prevents users from sending/receiving emails, etc.
- High: Bugs that are related to the core functionality of the application, but don’t have to be fixed before product launch. However, these bugs should be fixed in the first available patch or release after launch.
Examples: A flash demo doesn’t load properly, a bug tracking application that does not allow users to set a bug type/severity, a typo in the large text of the homepage banner, etc.
- Medium: Bugs that do not affect any critical user functionality. Typically, medium severity bugs have workarounds that allow users to accomplish the desired task that the bug may have hindered or the function may still operate but in a degraded fashion.
Examples: A travel site that randomly fails to send email notifications, search results page that displays the right results, but formats incorrectly in Chrome browser etc.
- Low: Bugs that do not interfere with core functionality and are just annoyances that may or may not ever be fixed.
Examples: A misspelling in the middle of an interior page, text being displayed in the wrong location, the sitemap loads improperly in IE6.
Understandably, the testing manager and customer are more familiar with the tested software and reserve the right to re-classify bug types. We ask both our customers and testers to respect the decisions of classification and re-classification, and ultimately be fair with these decisions.
To discuss this topic on the forums, please click here.
Bug Review Process
The bug approval process is as follows (similar with all other reports including test cases, surveys, etc.):
- Tester submits a bug in a test cycle (after accepting the Test Cycle Agreement on an Active test cycle)
- The bug moves into the “New” status, and on test cycles with Test Team Leads, your bug may be pushed to one of two other states called “Pending Approval” or “Pending Rejection.” This simply means that your bug has been recommended to the customer one way or the other by a Test Team Lead.
- Depending on the customer’s preference, the bug may be reviewed while the test cycle is still active or after it becomes locked. (Most bugs are reviewed within one week of the test cycle becoming locked.)
- Once the customer reviews a bug, the bug may be moved into the following statuses:
- Approved: Bugs are approved at either Exceptionally, Very, or Somewhat value and the payment is credited into the tester’s pending payments.
- If your bug is approved and the value reads ‘Not Specified‘ this means that the test cycle was closed automatically by the system and will be paid out as though it were a ‘Somewhat’ valuable bug. The ratings impact however, will fall between ‘Somewhat’ and ‘Very’. Typically this means that the customer or Project Manager has gone in and selected all of the very/exceptional bugs and set the rest to auto-approve once the cycle closes, leaving them as ‘Not Specified’.
- Info Requested: Customer has requested more information from tester. The tester needs to review the customer’s comments, provide additional information, and click on the “Send Requested Info” button for the bug to be moved back into the “New” status. Important Note: simply replying to the Tester Messenger will not push your bug back into the “New” status. If you do not take action to move your bug back into the “New” status, you risk having your bug rejected when the test cycles closes.
- Rejected: Bug is rejected by the customer. When a bug is rejected, the tester may choose to dispute the rejection. However, each tester has a limited number of disputes per month, so please choose which rejected bugs to disputes wisely. Dispute statuses include:
- Disputed: The tester decides to dispute a rejected bug and has one chance to provide further evidence of the bug
- Under Review: The customer has rejected a disputed bug and will not review any further evidence. However, the project manager may review, depending on the test cycle schedule and end date. If a report remains in the “Under Review” status when the test cycle closes, the report will be eventually marked as “Rejected” (you will also receive an email with Reason listed as “Other” and Explanation listed as “Test cycle closed”).
FAQ: How long do I need to wait until my bug is reviewed? (Similar with all other reports – including test cases, surveys, etc.)
- The review timeframe will vary. There are some customers who consistently review reported bugs as they trickle in. For example, the test cycle may still be active, but the customer is already approving or rejecting bugs as they are being reported. On the other hand, there are also customers who wait until the test cycle is complete before reviewing the reported bugs. As a general rule of thumb, please allow 35 days after the test cycle locks before sending a request about why your report has not been reviewed yet. You may request this information by emailing email@example.com.
Check for Duplicate Bugs Before Reporting
Every test cycle will have a list of bugs submitted by all testers. To view the bugs list for a particular test cycle, click on the test cycle and then on the Issues Tab. Alternatively, when you begin to file a bug, the platform will automatically search for keywords within the cycle’s Issue Reports and prompt you with a list of possible duplicates for review.
In addition to previously reported bugs, test cycles may also include a ‘Known Bugs’ List as an attachment in the Scope & Instructions tab. Please check these two lists before reporting a new bug.
FAQ: Are duplicate bugs accepted?
- In the case where there are duplicate bugs submitted, the first tester to submit the bug will be paid. All other testers will have their duplicate bugs rejected, so please check the known bugs list and previously submitted bugs before submitting a bug.
- There may be times in which it is difficult to understand the nature of a reported bug by simply reading through the bug title and bug report. In these cases, it is recommended that you click on the bug report attachments to view any relevant screenshots and/or video captures. The attachment viewer allows you to view these screenshots and video captures without the need to download the files to your local machine.
FAQ: Can I submit bugs that were approved in a previous test cycle?
- Unless otherwise noted by the Project Manager, please do not report bugs that were approved in a previous cycle, even if the issue is still occurring. Since the issue has already been reported and presented to the customer, reporting the same issue again does not provide any additional value. Keep in mind that if the customer wants to validate that a reported issue is fixed they will utilize uTest’s Bug Fix Verification services.
Bug Report Integrity
Most testers understand the role of a bug report is to provide information, however a “good” or valuable bug report takes that a step further and provides useful and actionable information in an efficient way. As such, in addition to approving tester issues, TTLs and Project Managers have the ability to rate the integrity of a testers bug report by setting the bug report integrity to High, Unrated or Low (by default, all bugs will set to Unrated).
The Bug Report Integrity feature will reward testers who meet a high report integrity standard by providing a positive rating impact to the quality sub-rating. Conversely we will also seek to educate testers who may be missing the mark by negating any positive impact that may have occurred based on the value of the bug itself. As such, each week testers will receive an email with a list of all bugs which have been given a bug report integrity rating.
The art of creating a well-written bug report requires a balanced combination of testing and communication skills. For more information on the standards and guidelines by which we evaluate the integrity of a report please review the following uTest University course: How to Write a Quality Bug Report.
NOTE: Bug Report Integrity has no impact on your payment.
Bug Reporting FAQ
We have compiled some frequently asked questions regarding situations encountered after a bug report has already been successfully submitted.
Who accepts and rejects bug reports?
- Most of the time, the testing manager (customer) reviews all of the bugs and accepts or rejects as necessary. Other times, the project manager (uTest) will review the list for obvious duplicate bugs or those that require additional information.
I was asked for more information on my bug, how do I document and prove that a bug is real?
- Provide relevant attachments such as screenshots and video captures. Please upload these video captures directly to the uTest Platform. Many more tools and resources are available on the Forums under Useful Testing Tools and Applications.
- This means that the customer has requested more information from the tester. You will need to review the customer’s comments, provide additional information, and click on the “Send Requested Info” button for the bug to be moved back into the “New” status.
- Important Note: simply replying to the Tester Messenger will not push your bug back into the “New” status. If you do not take action to move your bug back into the “New” status, you risk having your bug rejected when the test cycles closes.
When should I discard a bug?
- Prior to submitting a bug, all testers should carefully ensure that the bug is not a known or previously submitted bug. To verify your bug against known bugs, click on the “Known Bugs” document under Scope & Instructions (if provided). To verify your bug against previously submitted bugs, click on “Bugs and Test Cases” and filter bugs by the specific test cycle you are submitting bugs for.
- However, if you do in fact submit a bug that has already been submitted or perhaps you realize that it is not a bug, you have the ability to discard your own previously submitted bug . In other words, if the customer or project manager has not initiated a Tester Messenger conversation, or approved/rejected your bug yet, you may remove your bug by clicking on the bug report and then clicking on the Discard button to delete your bug (the Discard button is located on the individual bug report page, immediately above the Tester Messenger). This will remove your bug permanently and avoid potential rejections due to carelessness or oversight.
Is there a limit to the number of bugs I can report?
- If you are new to uTest and have not submitted any bugs yet, you will start out with a limit of five unapproved bugs per test cycle. When one of your bugs gets approved, you will be able to submit one more bug; so on and so forth. If you perform well in your first few projects, this restriction may be lifted. Specifically, if you are a Gold, Silver, Bronze or Proven rated tester, you will not have a bug limit per test cycle. Maintaining a high bug approval percentage and participation level will ensure that you maintain your Gold/Silver/Bronze/Proven status, thereby eliminating the five unapproved bugs limit.
- If you do not have a badge, be especially cautious for your next several test cycles and submit bugs that you are fairly certain will be approved (provide relevant attachments where applicable). Focus on increasing your overall bug approval percentage so that your rating will increase. This will not only benefit you by being able to submit more bugs per test cycle, but also by increasing the number of invitations to private test cycles
- Remember, test cycles may lock well before the stated “End Date” listed in the platform. Therefore, if you’re able to participate in a test cycle, please do so as early as possible, submitting in-scope findings sooner rather than later. When a test cycle locks, it may not reactivate again, so do not count on submitting any bugs unless the customer or project manager mentions otherwise in the Real-Time Chat.
Can I submit a “placeholder” bug report?
- No. When you submit a bug report, finalize all contents of your report before clicking on the “submit” button. Although you may return to your bug to make minor edits, you must not make major edits to your report – including bug title, actions performed, expected results, and actual results. The reason this practice is prohibited is because it severely hampers fellow testers’ ability to spot duplicate bug submissions (duplicate bugs are rejected). If you happen to observe other testers creating placeholders, please send an email to firstname.lastname@example.org with relevant screenshots of this violation.
I started to report a bug, but the test cycle became locked. What can I do from here?
- If a test cycle becomes locked, you cannot submit any bugs or test cases (even if you are in the middle of submitting a new bug). You can wait for the test cycle to reactivate again or wait for the next one to activate (if applicable). Remember, only bugs and test cases that are submitted through the uTest platform will be considered for approval and payment.
Why is the “File Bug Report” button disabled?
- First, you need to make sure that the test cycle is marked “Active” in order to participate. Next, click the “Accept Test Cycle” button under the Test Cycle Agreement if you are willing and able to perform testing during the specified time. If you performed both of the tasks above, there may still be instances where the “File Bug Report” button is disabled. For example, the customer may only be looking for test case results and has thereby disabled the ability to report any bugs. The opposite case is also true; the customer may only be looking for bugs as opposed to test case results. Also, there are times when bug reports cannot be submitted against an active test cycle, especially after unusual spikes in participation. Please standby and wait for further announcements from the project manager before attempting to submit subsequent bug reports. Finally, there is one more scenario when you may be unable to submit any more bugs and/or test case results: Your tester rating is below the Proven tier. If this is the case, you have a limit of five unapproved bugs per test cycle.
Disputing bug reports can be a delicate situation, and here you can learn how an ideal dispute should be conducted, what to avoid during a dispute, and when to avoid a dispute at all. Eventually most testers will run into a situation like this, but they can be opportunities to educate others with constructive concepts and expanded evidence.
Dispute a Rejected Bug:
When you should or should not dispute a bug
You should only dispute a rejected bug if you are certain that you are correct. Perhaps you did not elaborate enough in your first attempt, and now have additional information for the customer to properly understand the bug. You may use the Real-Time Chat to clarify any misunderstandings centered around the scope of the test cycle (do not use the Real-Time Chat for difference of opinions on how things should work or post a bug to the Real-Time Chat to request if it is valid or not).
If you determine that you need to dispute the bug, please click on your rejected bug and then click the Dispute Rejection button; you will be provided one chance to provide more information to the customer for further review. Be sure to communicate professionally and provide further evidence, such as screenshots and video captures (you can upload new attachments after you dispute the bug by clicking on the “Edit” button of the report). This will increase the likelihood of your dispute to be accepted.
One note of caution: Do not overuse this feature. If your bug is a duplicate, out of scope, or not reproducible, it may be rejected completely – even if you dispute it. On the other hand, if you were misunderstood, then you may re-explain concisely (you will have limited space for your dispute explanation). Additionally, never ask the customer to approve your testing work (this includes requesting them to accept as feedback). Testers who violate these rules may be subject to temporary or permanent account suspensions.
Additionally, do not use the Tester Messenger or Dispute feature to negotiate with the customer in any way, In other words, do not ask the customer not to reject your bug or bring up your tester rating as a reason for not rejecting the bug. We monitor all disputes and tester messenger discussions; testers in violation of this rule will be subject to further review and possible suspension of their accounts.
We recommend that you so that you will not need to dispute a valid bug. And because testers are allotted a limited number of total disputes per month, we advise that you use this feature sparingly and accordingly.
We have compiled some frequently asked questions’ regarding situations encountered after a bug has been rejected and a tester desires to dispute it. Remember, you may only use the dispute feature to dispute a bug – do not use the Real-Time Chat or Tester Messenger to dispute.
If my bug gets rejected, how do I find out the reason?
- When a bug is rejected, an email containing the rejection details is automatically sent to the tester. Please sign in to the uTest platform to follow up with bug disputes as needed.
Can I dispute a bug for any reason?
- Testers may dispute rejected bugs, so long as you are fair and communicate professionally with customers. Note: All testers have a limited number of disputes per month, so please use them appropriately. If you decide to dispute the bug, be sure to provide further evidence, such as screenshots and video captures and upload these directly to the uTest Platform (you can upload new attachments after you dispute the bug by clicking on the “Edit” button of the report). This will increase the likelihood of your dispute being accepted.
Can I upload new attachments when I dispute a rejected bug?
- Yes. In fact, we recommend that you always provide further evidence upon disputing a rejected bug. This could come in the form of new screenshots or even a video capture. Please note that you can only edit your bug report and upload new attachments after you have disputed your report.
I cannot dispute any more bugs. Why?
- All testers have a limited number of disputes per month. Each tester has 2 disputes, plus 10% of his/her approved bug count in a 30-day window. If a tester’s dispute is approved, the dispute approved bug will not count towards the dispute limit.
What if my dispute gets rejected but I still disagree?
- If your dispute is rejected, there is no further action you can take to reverse the customer’s decision. In order to avoid this, please make sure that your bug report is clear, concise and within scope. This is by far the best way to avoid bug rejections and disputes.
Test Case Submission and FAQ
If a test cycle contains test cases to be completed, you will first need to claim them from the test case library. To do so, accept the test cycle agreement and proceed to “My Test Cases.” On this screen, you will be presented with the test cases associated with this particular test cycle. Under “Available Test Cases,” you can view which test cases are available for you to claim based on your testing environment and predetermined rules set by the customer (e.g. limited number of testers per environment, limited number of pending test cases per tester). If a test case is available for you to claim, you will see a “Claim” button under the ‘Action’ column; if unavailable, you will see “Not Claimable” instead. To preview each test case, click on the test case name. To claim an available test case, click on the ‘Claim’ button to the right of each test case. When you claim a test case, you will need to select the testing environments you are able to test against. Only claim one testing environment per test case unless otherwise noted under ‘Scope and Instructions’.
Remember, you may not claim any test cases once you have reached the predetermined limit per tester and/or the limit of pending test cases per tester (this limit is based on the test case level, as opposed to the test case and testing environment combination level). Once you have completed one or more test cases, check back under “My Test Cases” to see if you’re eligible to claim more test cases. Additionally, you can still claim test cases if the test cycle is in the locked status, but you cannot submit results unless the test cycle reactivates. When you are ready to begin executing the test case, please click on the “Start” button of the relevant test case. You then have the ability to click Pass/Fail next to each step until all steps have been completed. If you are unable to complete the steps, you may click the “Submit” button and select a reason (e.g. invalid test case, unable to complete). Doing so will result in a “Skipped” test case, and no payments will be issued. After you have submitted your test case, you may click on ‘My Test Cases’ at any time to view the status of your submitted test case (approved/rejected).
Test Case Submission FAQ:
What does the Action column status ‘No Environment Matches Remain’ mean?
- This means that all available test cases have been claimed in this test cycle. If this is the case, check to see if the test cycle is ‘Exploratory with test cases’ in which case you can use the URL to report bugs as opposed to submitting a test case.
How many test cases can I claim?
- Only accept one testing environment per test case unless otherwise noted. Additionally, there is a limited number of pending test cases you can claim across all of the test cycles you’re currently participating in.
What if a test case step fails?
- If one or more steps fail, complete the remaining steps and submit when all steps have been marked pass or fail. Additionally, you have the option of filing a bug report (dropdown menu on the Fail button). Finally, the submitted test case will be marked Fail
Can I change the status of my submitted test case (from pass to fail or vice versa)?
- Yes. If you need to revise the pass/fail status of one or more test case steps, click on your submitted test case to find the “Undo Submission” link (next to the Discard button). When you click on this link, you will be able to change the pass/fail status for each step by clicking on the “X” next to the Passed/Failed image. After you make changes to relevant test case steps, click on the Submit button to commit to your changes and put your test case back in Pending Approval Status.
Can I attach a file to my test case?
- You may upload it to your test case under “Result Attachments” in the test case itself.
Can I discard or unclaim an entire test case result?
- Yes. To unclaim a test case, click on the test case in question. Next, click on the “Unable to complete” link to the left of the Start button. Finally, select “Unclaim” option to permanently unclaim and discard your test case. Please note that you will not be able to retrieve your test case once you have discarded it.
Why does my test case have a status of STARTED or CLOSED BY TEST CYCLE?
- If you claim and begin a test case while the test cycle is still active or locked, the test case status will be labeled STARTED. If you do not complete the test case before the test cycle closes, the test case status will then be labeled CLOSED BY TEST CYCLE. In these cases, your unsubmitted test case(s) will not be approved nor paid out.