Repomaker Usability, Trainers Worldwide Study
Prepared by Carrie Winfrey and Tiffany Robertson, Okthanks, in partnership with F-Droid and Guardian Project
For more information, contact firstname.lastname@example.org.
The purpose of this study was to understand the following things.
- Are users able to complete basic tasks including, creating a repo, adding apps from other repos, removing apps, editing app details, and creating a second repo?
- Do participants understand how to get the apps from a repo installed on an Android phone?
- Word choice—Do people understand the word repo?
- Is repomaker a useful tool to participants?
The study was hosted via video calls with potential users within our target audience. It was focused on the trainer use case. The narrative below describes the target use case.
Narrative. You live or teach people in a location with poor internet quality. Your wifi and data connection are not reliable, and rarely can you use Google Play to get apps and app updates on your phone. This makes it difficult to do your job. As an instructor, your job is to train journalists how to stay safe while using their phones. You teach them ways to keep their communications and media private.
The study consisted of 3 parts.
- Usability study. Participants were asked to use Repomaker to complete tasks as we prompted them to do so. After completing the tasks, participants were asked to rate its level of difficulty.
- Login to the app. Create a repo and add three apps that are useful for the journalists you train.
- Add the Twitter app.
- Add the Courier app from the Guardian Project’s repo.
- Add a description to the Courier app.
- Remove an app from your repo.
- Create a second repo. You don’t need to add any apps.
- Reopen your first repo.
- Share your first repo with your journalists.
- Open the page you just shared.
Talk through how your journalists would add this repo to their android phone.
Debrief. The following questions were discussed.
- Would the individuals you train feel comfortable downloading F-Droid in order to get access to the apps in a repo? (Note: F-droid cannot be downloaded from the Google Play store. It can be downloaded directly from the fdroid.org website or received over bluetooth from someone in the same room. Also, downloading F-Droid requires you to enable unknown sources on your phone.)
- Please tell me how Repomaker would be useful for you?
Survey. Users were asked to complete the survey immediately after the video call:https://goo.gl/forms/Acq4zYB3KOtQPxBa2
What was tested: Link to youtube video.
9 people participated in the study from countries including Guyana, India, Pakistan, United States and Zimbabwe. 50% of have experience training. View Demographic stats
Participants were thanked with a $20 gift card to Amazon or Google Play.
In general, people are excited about this tool. A majority of the participants feel that the interface is straight-forward and easy to use. 8 of 9 participants felt that Repomaker is a useful tool and would use it. It is most useful for distributing apps in cases where there is no internet connection. See notable survey results Biggest Challenge.
- Users need a core understanding of F-Droid and how it works
The majority of participants felt that they would need to understand F-Droid first before using Repomaker. More info in task 10
- We need to talk about F-Droid in a way that is familiar to non-technical users.
- For example: F-Droid is an independent, community-sourced app store. You can customize it by adding repos, collections of apps shared by sources you trust.
- Make a strong brand connection between Repomaker and F-Droid. Users need to understand that Repomaker is part of F-Droid.
- Provide step-by-step instructions for installing F-Droid and installing apps from a repo on an Android phone.
- Consider the touchpoints for a first time user—
- Goes to the F-Droid website
- Views Repomaker page (on the F-Droid website)
- Launches Repomaker
- Opens in a new tab to Login/Signup view.
- Select Sign Up. Enter info. Action button says “Create my first repo”
- New repo view
- Single repo empty view
2. Repo is an abstract concept
- Most participants understood this after they started using the app. Link to survey result
Though, upon first use, many participants tapped the ‘Add’ button instead of ‘Add from repos’ because they were unsure what ‘Add from repos’ meant. More info in task 1
- Use the wording “Browse gallery” instead of “Add from repos”.
- Indicate how much stuff is there (2,000 apps, …)
Use less words in the single repo empty state view. **TICKET #120
- Provide blurbs about what a repo and and what they can be used for.
3. Participants had questions and concerns about editing
- Many participants were surprised they could edit apps from other people and commented that they wouldn’t have known they could do it if we hadn’t asked them to do so.
- One participant was unsure if the editing function worked like a Wiki and would update everywhere, or if it would only appear in her repo.
- One participant commented that liked that she could add her own translations.
- Some participants expressed their concern about the security of this feature.
- UI for editing is confusing. Description box takes a long time to load and requires scrolling to see.
- More info in task 4
- Address questions and concerns in a FAQ page
- Show bar on editing page, when editing a file from another repo. Ticket ..
- Allow the name and developer to be entered only when the first version of a file is added. Do not allow this to be changed afterward.
Add hints to summary and description field. **TICKET #103
- Provide ‘save’ buttons under text sections rather than one ‘done’ button for the entire page.
Offer translation option within Repomaker **TICKET #58
- If it comes from a remote repository AND there is text in the field, then make it read-only.
- Hook into FAQ about why you can edit things.
- Some participants shared concern about people being able to masquerade an app with the editing feature.
- One participant was unsure how public the repo was when she was creating it, and was hesitant about how she named it.
- One participant commented that she didn’t want to be responsible for sharing non-secure apps.
- Allow the name and developer to be entered only when the first version of a file is added. Do not allow this to be changed afterward.
- Provide clear information when a repo is created about who has access to the repo.
- Provide a clear ‘publish’ step, so that users know when I repo is made visible to others.
- Address concerns in a FAQ.
- It’s unclear that you’re searching within a category if you have one selected.
- Most participants commented that they wanted a clear way to go back from this view. Some would tap back in the browser and changes wouldn’t be saved.
- More info in task 2
Redesign how search and filters work together in the UI **TICKET #119
- Add apps immediately once selected. Offer the option to remove rather than being able to ‘uncheck’.
- Count apps when added.
- Critical Errors: Critical errors are reported as errors that result in failure to complete the task. Participants may or may not be aware that the task goal is incorrect or incomplete. Independent completion of the task is the goal; help from the test facilitator or others is to be marked as a critical error.
- Non-critical Errors: Non-critical errors are errors that the participant recovers from alone and are not such that the participant can no longer complete the task. They can include errors such as excessive steps taken to complete a task or initially using an incorrect function but recovering from that incorrect step.
- No Problem: A qualitative analysis of users ability to complete the task with seemingly ‘no problems’. However, this does not mean that non-critical errors were not made while completing the task.
- Task 1: Create a repo and add 3 apps.
- Users becoming familiar with interface also many believed once they hit ‘add’ and saw the blue button their app had been added or saved to their repo.
- Task 3: Add the Courier app from the Guardian Project’s repo.
- Rated lowest level of difficulty because users have become familiar with the interface.
- Task 5: Remove an app.
- Rated highest level of difficulty because users felt there were too many steps involved and most expected to be able to remove an app from the repos view.
- Task 6: Create a second repo.
- Task 8: Share your first repo with the journalist you train.
Task 1: One participant unable to complete the task without help from the facilitator.
- Participant thought she could add anything from her applications folder on her computer.
Task 3: One participant unable to compete the task without her from the facilitator.
- The Courier app was already in their repo and they didn’t see the error so when clicking the ‘done’ button they were not brought back to their repo.
Task 4: Two participants unable to complete the task.
- Added the description to the summary field only (instead of the correct text box).
Task 5: One participant unable to complete the task.
- Opened and closed app index and app details twice.
- Couldn’t figure it out from the repo view and didn’t choose ‘edit’ in the app details view because they weren’t making edits to the app just the repo.
Task 8: One person was unable to complete the task.
- They tried to get the repo added to their phone first, but was technically unable to do so because of storage limitations.
4. People have concerns about the security of Repomaker
5. UI for search and filters in the ‘Other Repo/Browse’ view is unclear
Usability metrics refers to participant performance when completing the assigned tasks. This is includes completion success rates, error rates, time to task completion and subjective evaluations/interviews.
Level of Difficulty:
The participants were asked to rate the level of difficulty per task as they completed them.
Average Critical Error Rate:
Critical errors are reported as errors that result in failure to complete the task. Participants may or may not be aware that the task goal is incorrect or incomplete. Independent completion of the task is the goal. Help from the test facilitator or others is to be marked as a critical error.
<p> <b>Average Non-critical Error Rate:</b> </p> <p> Non-critical errors are errors that the participant recovers from alone and are not such that the participant can no longer complete the task. They can include errors such as excessive steps taken to complete a task or initially using an incorrect function but recovering from that incorrect step. </p>
- Task 1 & 2: Participants learning the interface and multiple steps required to complete the task.
- Unfamiliar with the word repo.
- Were distracted by the ‘add’ button on repo page.
- Issues with adding from other repos-See details in tasks 1 & 2.
- Blue check didn’t conform to their expectations.
- Task 5: Too many steps involved. Many thought this action would be found within the repo view.
- Task 8: Participants tapped the ‘copy link’ button and after seeing the blue check thought they were finished, but not confident in what just happened. Afterwards, copying and pasting the link in a new tab.
Task Specific Findings
Task 1: Login to the app. Create a repo and add three apps that are useful for the journalists you train.
- Average Difficulty: 1.83
- Number of Participants with Critical Errors: 1
- No problem – 4⁄9 (44%)
“I wouldn’t actually put the country name, because I’m not sure how public this is.” (Participant 9)
“I don’t want to be responsible for sending out non-secure apps.”
Apps Index View – Before Apps are Added
- Clicked ‘add’ first – 4⁄9 (44%)
- Thought the ‘+’ in the nav bar was to add apps – 2⁄9 (22%)
- Unclear that this was only for Android apps; Was confused about which apps could be added; thought she could add things from her applications folder on her computer – 1⁄9 (11%)
- Confused about adding .apk files – 1⁄9 (11%)
Add from Repos View
- Thought when they tapped ‘add’ it would add the app, but it didn’t until they tapped done
- The problem was that people went to the next page or changed the filter and their selections were cleared- 3⁄9 (33%)
- Users noticed that some apps showed up multiple times – 2⁄9 (22%)
- Some left the page without hitting done (ie. used the back button in the browser) – 1⁄9 (11%)
- Wanted to sign up using a google account – 1⁄9 (11%)
- Participants glad to see the apps they wanted are available
- Be specific about what file types can be added
- “Can I add my own files?” (Participant 9)
- Create buttons appears to ‘lag’ – 7⁄9 or (77%)
- How do I ensure my repo is private?
Task 2: Add the Twitter app.
- Average Difficulty: 1.55
- Number of Participants with Critical Errors: 0
- No problem – 4⁄9 (44%)
<p style="text-align:center;"> <i>“I don’t know what ‘other repos’ means.” </i> </p> <p style="text-align:center;"> <i>(Participant 9)</i> </p> <p style="text-align:center;"> <i></i> </p> #### Observations: Apps Index View – After (task 2) * Big shiny add button was distracting; thought they would click there to add more apps – 2/9 (22%) * Were confused or didn’t like that the initial action cards merged – 4/9 (44%) #### Participant Feedback: * ‘Add from repos’ button needs to be more visual and bigger * Thought the explore icon looked like an eyeball <p style="text-align:center;"> <p style="text-align:center;"> <i>“I wasn’t sure how to get back to the page I was on.” </i> </p> <p style="text-align:center;"> <i>(22% of Participants)</i> </p> <p style="text-align:center;"> <i></i> </p> <p style="text-align:center;"> <i></i> </p> <p style="text-align:center;"> <i></i> </p> </p> Add from Repos View * Need better categories (ex: ‘internet’ category is a little abstract – includes browser’s, privacy twitter and facebook) A social media category would be more relevant __– 2/9 (22%) * It’s not immediately clear that categories filter – 2/9 (22%) * Participant chose a category filter, then searched. The app didn’t show up, because it was searching within the category. * Search was not easy to find- 1/9 (11%) #### Participant Feedback: * “Categories aren’t specifically clear.” (Participant 1) * _“I’m used to finding search on the right side.” (Participant 7)_ #### Bugs: * Bug filed for adding an app that is already in your repo #108 </p> ## ## PAGE_BREAK: PageBreak ## Task 3: Add the Courier app from the Guardian Project’s repo. * **Average Difficulty**: 1.22 * **Number of Participants with Critical Errors**: 1 * **No problem** – 8/9 (88%)
#### Observations: * Participants familiar enough with interface to easily complete task #### Participant Feedback: * One participant made 7 redundant taps on the ‘done’ button during this task. They assumed the page was stalling or the internet connection was unreliable. When in fact, they already had the app in their repo and didn’t see the error/warning message at the top of the page. Wasn’t able to get out of this view unless they reloaded the page or unchecked the blue check mark. </p> ## ## PAGE_BREAK: PageBreak ## Task 4: Add a description to the Courier app. * **Average Difficulty**: 1.5 * **Number of Participants with Critical Errors**: 2 * **No problem**– 5/9 (55%)
<p style="text-align:center;"> <i>“It seems really strange to be able to edit someone else’s app. I’m really concerned about the security feature of publishing repos. People putting out virtually similar apps, but changing just slightly. Because I can edit almost everything except for the icon. I can upload my own app file but still use their icon. Suggest ways to have warnings about masquerading.” </i> </p> <p style="text-align:center;"> <i>(Participant 1) </i> </p> <p style="text-align:center;"> <i></i> </p> <p style="text-align:center;"> <i>“Does it work like a wiki when I make changes, or does it only change it in my repo?” </i> </p> <p style="text-align:center;"> <i>(Participant 5)</i> </p> <p style="text-align:center;"> <i></i> </p> #### Observations: * Added description to the summary field – 3/9 (33%) * Commented that they wouldn’t have known they could edit an app, unless we asked them to do it – 3/9 (33%) * Description view takes too long to load – 2/9 (22%) * No indication in the description text box that this is where you add a description – 2/9 (22%) * Want a ‘save’ button with the description box – 2/9 (22%) * Want a back button on this page (wants it to say ‘back to repo’) – 2/9 (22%) __ #### Participant Feedback: * If you tap back in the browser, your changes are not saved. * Attach save buttons to sections that are edited, rather than having 1 ‘done’ button * Talk about editing in the FAQ or overview of the app * Simplify edit view</p> ## Task 5: Remove an app from your repo. * **Average Difficulty**: 2.16 * **Number of Participants with Critical Errors**: 1 * **No problem** – 4/9 (44%)
<p style="text-align:center;"> <i>“Edit makes me think that I would edit something within it, rather than removing it.” </i> </p> <p style="text-align:center;"> (Participant 3) </p> #### Observations * Expected to be able to remove apps from the main apps index view – 7/9 (77%) * Most people searched main page, but remembered that they had seen something in the edit view previously. * _I’m not making any changes to the app._ * _Wasn’t immediately apparent_ * _Not difficult, just requires a lot of clicks_ * I would expect to hover over or long press an app and get the option to remove * I want a ‘remove app’ button on the app page * Likes confirmation to remove the app – 1/9 (11%)</p> ## Task 6: Create a second repo. You don’t need to add any apps. * **Average Difficulty**: 1.72 * **Number of Participants with Critical Errors**: 0 * **No problem**– 6/9 (66%) ####<img src="https://guardianproject.info/wp-content/uploads/2017/06/image5.png" width="740" height="333" alt="" title="" /> #### Observations: * Not completely clear, but figured it out – 3/9 (33%) * Trying to find empty state of My Repo view – 2/9 (22%) * Many had tapped + by mistake already (thought it was to add apps)- 2/9 (22%) * Wanted to tap on the name of the repo or near it – 1/9 (11%) Participant Feedback: * Would help to have the option in the My Repos home view * Hover label for new repo (near +) * “Add button says it all.” (Participant 4) * Would like a visual notification that the first repo was saved ## ## Task 7: Reopen your first repo. * **Average Difficulty**: 1.44 * **Number of Participants with Critical Errors**: 0 * **No problem** – 8/9 (88%)
<p style="text-align:center;"> <i>“Home conformed to my expectations.” </i> </p> <p style="text-align:center;"> (Paricipant 5) </p> #### Observations: * Had figured out how to do it in a previous task – 1/9 (11%) * Tapped back in browser – 1/9 (11%) #### Bug: * Bug with duplicate repos appearing in home screen 1/9 (11%)</p> ## ## Task 8: Share your first repo with your journalists. * **Average Difficulty**: 1.72 * **Number of Participants with Critical Errors**: 1 * **No problem**– 3/9 (33%)<img src="https://guardianproject.info/wp-content/uploads/2017/06/image7.png" width="716" height="305" alt="" title="" /> #### #### Observations: * Copy and pasted the link in a new tab (without prompting) – 4/9 (44%) * Went to share right away – most everyone * Participants tapped ‘copy link,’ saw a blue check and did nothing else – 2/9 (22%) * Used Facebook – 1/9 (11%) * Used Twitter – 1/9 (11%) * Opened the public link right away. Was going to add it to F-Droid then share with people from there 1/9 (11%) * Not clear how to publish the repo – 1/9 (11%) #### Participant Feedback: * Wants to make sure the link is secure * Don’t want to make it public (Password protection, unlisted?) * Check doesn’t go away after copying link * _“Sending a QR or link doesn’t do anything.” (Participant 1)_ * “What does the blue check mean?” (22% of Participants) * Would be nice to be able to email from this page, maybe linkedin * Like the QR code, because very familiar using these * Steps aren’t extremely clear * Likes that it shows how many apps are in your repo * Step by step — this is how you share it with journalists (3 step process) * Having the QR code and link on the same page is helpful * Would like to see a reminder about F-Droid on the share screen * Need to have a clear understanding about F-Droid and how it works ## ## Task 9: Open the page you just shared. * **Average Difficulty**: 1 * **Number of Participants with Critical Errors**: 0 * **No problem**– 9/9 (100%)<img src="https://guardianproject.info/wp-content/uploads/2017/06/image8.png" width="704" height="320" alt="" title="" /> #### Observations: * Everyone highlighted link, copied and pasted it in a new tab to view </p> ## ## PAGE_BREAK: PageBreak ## Task 10: Talk through how your journalists would add this repo to their android phone. #### Main Takeaways:</p> 1. **The majority of participants felt that they needed a core understanding of F-Droid first** __ _“Most people wouldn’t have a problem installing F-Droid. The biggest challenge is knowing how F-Droid works.” (Participant 3)_ __ _“The concept of a repo is very abstract – the hardest part would be understanding what it is.” (Participant 6)_ __ * First need to understand F-Droid in order to install a repo * Understood that you need F-Droid first, but believe you would need to spend time explaining FDroid to someone before setting up a repo * Relationship between F-Droid and repomaker is unclear * Might be a challenge if F-Droid is not already adopted into their workflow * People need to be comfortable and familiar with F-Droid first in trainings * If they are educated about F-Droid and info comes from a trusted source, they will be likely to use it * How do I get it if I only have one device 1. **Step-by-step instructions needed (beginning with F-Droid installation)** __ “Most of the journalists I work with are lazy. They would need step by step instructions.” (Participant 8)</p> * Step-by- step instructions would help * If I share it on Facebook, they will have it on their phone. * Provide info up front that you need F-Droid for it to work * QR code scanner in F-Droid would be helpful. Participant didn’t have a QR code scanner or room to download it * The Pakistani testers liked QR code option for sharing * Wants to make sure the link is secure </p> # # PAGE_BREAK: PageBreak # Is it Useful? <p style="text-align:center;"> 8 out of 9 participants agreed Repomaker is a useful tool and would utilize it </p> ##### YES * Yes! Participant does multiple trainings in rural areas with students who couldn’t download apps prior to trainings, making talking about the specific apps during the actual trainings too abstract. * Repomaker is a great idea! Very, very useful! * See this being a very useful tool for trainings. Easily send a link to trainees prior to the training/conference ask them to install FDroid and download this repo with all the materials needed for our training. * In a context where people don’t have Google Play, F-Droid makes a lot of sense. * Yes, because when teaching labs (at university) we must download a few network simulators and apps, but because of the proxies network (restrictions on the internet, sites or apps being blocked) we aren’t able to use wifi in this way, making it difficult or impossible to get the needed materials for lab. Therefore, we could utilize Repomaker in these instances (Nearby tool or bluetooth) * Some people would find it very useful! * Very useful to host trainings, give people a landing page with a customized repo! * If there is some prior education about FDroid and Repomaker and this education/introduction comes from a trusted source, believes many people would not hesitate to use the tool. * Participant is really struggling with the problem of unreliable internet connectivity, so they are very interested in this app specifically for training purposes and the ability to share repos with people whenever needed. ##### NO OR MAYBE * Not sure, I’m not an Android user. * How does this make my use case easier? * Want to know more about F-Droid ##### COUNTRY-SPECIFIC POINTS #### Pakistan * In Pakistan everyone uses Android. * In Pakistan mobile data is expensive and not all areas have good coverage, however, about 70-90% of people have mobile phones with airdrop being a very familiar sharing tool. * In big cities within Pakistan people are very familiar and comfortable using QR scanners and codes, however, within the smaller, rural areas people are skeptical. * Access to mobile phones ubiquitous, so mobile platforms really useful. #### Sri Lanka * In Sri Lanka, physical and device checks (army checkpoints) are very common. The guards will check your phone apps and photos, but people would be storing these repos in F-Droid, so apps wouldn’t appear on their home screen. * F-Droid doesn’t look attractive, so it won’t attract officials attention. * The galaxy y doesn’t have very much capacity. Storage space is an issue. * If there’s patchy network, people use bluetooth, to save data, because data is money and also to save data. * Most people are using phones that are given by organizations—like World Vision. Some organizations feel that it can be used as a personal device, so they block things so they won’t use data (ex: some block facebook or IMO apps). Sometimes they come with their own apps (like World Nation); blocking other apps. #### Zimbabwe * Challenge of internet very real in Zimbabwe. Repomaker is very practical and useful. Being able to store apps all in one location. * Too many people using Google Play at once. #### Participant Feedback: * Could offer a FAQ’s section * On home screen have a few quick questions and answers regarding Repomaker– _What’s a repo? What would I use repomaker for? Learn more here.</li> * Additional verification step? * How much will it cost to run the apps? * How much data will repomaker require? * What about storage? Will repos be able to store on external storage devices?</ul> </p> # # PAGE_BREAK: PageBreak # Survey Results</p> ## Follow-Up
## ## PAGE_BREAK: PageBreak ## App Distribution</p>
#### Notable: How many app? From our responses, a majority of trainers share 2-5 apps during trainings.</p>
####<img src="https://guardianproject.info/wp-content/uploads/2017/06/null18.png" width="624" height="340" alt="" title="" /> #### Notable: Biggest challenge? _**The biggest challenge this set of trainers face is that ‘Participants can’t connect to the internet during a training (75%).</i>**</p> _**Second to that (50%), ‘Participants can’t connect to the internet before the training’ and ‘Sharing an app requires unknown sources to be turned on in the phone settings’.</i>**</p> _**</i>**</p>
## ## PAGE_BREAK: PageBreak ## Basic Info</p>
## ## PAGE_BREAK: PageBreak ## Would you do it again?<img src="https://guardianproject.info/wp-content/uploads/2017/06/null28.png" width="624" height="324" alt="" title="" /> ###<img src="https://guardianproject.info/wp-content/uploads/2017/06/null29.png" width="624" height="278" alt="" title="" /> PAGE_BREAK: PageBreak # Possible FQA’s</p> How much data will repomaker require? How much data is required to install F-Droid? How much storage needed? Will repos be able to store on external storage devices? What is F-Droid? What is a repo? What files can I add to my repo? Can apps be downloaded directly from the public repo view? Why do I need F-droid? Can I edit any app? Can others edit my apps? Can others edit my personal repo? How can I customize to a specific training? How public is my repo? Who has access to my repo? How secure is my repo? How do I verify the apps I’m adding to my repo? How do I know apps are coming from the original source? How do I guard against malware? If there are two of the same app, how do I know which one to trust? Can other people masquerade my apps? Is it safe to enable unknown sources? How do I install apps from my repo? Is this a secure tool? Is it safe to use? How much storage will Repomaker and F-Droid need? Can the apps from my repo be downloaded onto an external storage device?</p> **How to use F-Droid Repos** – how to add one / how to get a repo on your phone – how to get a repo from the desktop computer to your phone – how to share a repo link – how to share a repo from F-Droid when there’s no internet; how to share one offline / how to share apps with peers when there’s no internet. – how to remove one – how do I update files and apps I add?