At Dutch Lotteries (Nederlandse Loterij) Sander uses the same Robot Framework repository with 11 other testers across 7 teams. Their approach can scale up further. Peek behind the curtain at how they maintain quality across all test suites. Building quality test automation at scale is hard, especially when many people of various skill levels are involved. Often, this results in clashing approaches, accidentally rebuilding existing functionality, bad code, and inconsistency.

At Dutch Lotteries (Nederlandse Loterij), they found various ways of dealing with scaling issues:

• Test Automation Guild to raise skill levels.
• Required reviews as a teaching tool.
• Build well-documented and flexible shared functionality.
• Structure the repository for clear scopes and flexibility.
• Write organization-specific standards.
• Make lint rules to enforce their standards, you’ll take a look at their approach so you can use those ideas to scale up your test automation.

References and further reading

• I write articles
• Robot Framework

At Dutch Lotteries

Dependencies used

Foundational libraries

• AppiumLibrary
• RequestsLibrary
• SeleniumLibrary

Other libraries

• CacheLibrary
• DatabaseLibrary
• FakerLibrary
• JSONLibrary

Tools

• Pabot
  Run tests in parallel
• Robotidy
  Auto formatter
• Robocop
  Linter
• Find Unused
  Find unused keyword, global variables, arguments, and return values
• MkDocs + Material for MkDocs
  Documentation website builder

Standards

• Keyword names
  Standards on how to name keywords
  1. Start with the module code
  2. No abbreviations
  3. Use action verb
  4. No ‘and’ or ‘or’ conjunctions between action verbs
  5. Name contains interface descriptor
  6. Prefer framework keywords
• Keyword arguments
  Standards on using keyword arguments
  1. No embedded arguments
  2. Max 3 arguments without default value
  3. No unspecified argument defaults
  4. Use simple argument defaults
  5. At least 1 argument
• Storing locators
  Standards on how to store and manage locators
  1. Locator variables
  2. Locator variable names
  3. Locator variable files
  4. Grouping locator variables
  5. Sorting locator variables
  6. Comments
  7. Split up locator variable files
• Writing locators
  Standards on how to write a locator.
  1. Locate with XPath
  2. Locate using the most unique values
  3. Don’t use meaningless HTML tags
  4. Don’t use the XPath function text()
  5. Nest locators with one parent
  6. Use indexes when locating multiple elements
  7. Avoid using locator templating
• Test labels
  Standards on using test labels to locate elements. Shared with developers.
  1. Use the data-test attribute
  2. Add to visible elements
  3. Add to component wrappers
  4. Add on request
  5. No test without test labels

Custom lint rules

Name	Short description	Example
NLO-I01 import-from-lower-abstraction-layer	An important rule in the abstraction layer setup is that you are never allowed to import from a lower level of abstraction. Doing this will turn the nicely structured layers into spaghetti.
NLO-I02 import-from-outside-abstraction-layer-setup	If you import from outside the abstraction layer setup, you lose all the benefits.
NLO-F01 folder-name-casing	Consistent folder names can solve a lot of headaches and help you guess how a file path is written.	~/my_folder/sub/folder
NLO-F02 file-name-casing	Consistent file names can solve a lot of headaches and help you guess how a file path is written.	~/test_file.robot
NLO-K01 missing-module-code-in-keyword-name	Every keyword name must start with a module code. These module codes are used to distinguish between keywords of different modules in the abstraction layer setup.	GEN My Keyword
NLO-K02 inconsistent-keyword-module-code	Every keyword name in a file must start with the same module code. In a single file, all module codes must be the same.	GEN My Keyword GEN Another Keyword
NLO-K03 missing-interface-descriptor-in-keyword-name	After the module code, every keyword must contain an interface descriptor.	LOT GUI Create player LOT API Create player
NLO-K04 missing-action-verb-in-keyword-name	After the interface descriptor, the keyword name must contain an imperative verb.	LOT API Create player LOT API Delete player
NLO-K05 undocumented-abbreviation-in-keyword-name	Avoid using abbreviations in your keywords outside of the module code and interface descriptor. If you use an abbreviation anyway, it must be explained in the documentation.	EJP GUI Click CTA
NLO-K06 no-verb-conjunction-in-keyword-name	Keyword names with conjoined verbs are often an indication of unclear keywords or keywords that try to do too many things.	EJP GUI Check and Close Notification Message
NLO-L01 locator-variable-outside-locator-file	All locator variables live inside dedecated locator variable files.
NLO-L02 locator-file-too-big	A large file can cause issues with tooling (for example: auto-formatting). To prevent these issues, you should split your locator variable file if it’s larger than 500 lines.
NLO-L03 locator-variable-name-prefix	The first part of every locator name must be the literal text loc.	${loc.EJP.someLocator}
NLO-L04 locator-variable-name-missing-module-code	The module code part of the locator refers to the module code of the current abstraction layer. The module code must be uppercase.	${loc.EJP.someLocator}
NLO-L05 locator-variable-name-inconsistent-module-code	The module code part of the locator refers to the module code of the current abstraction layer. Only a single module code can be used in each locator file.	${loc.EJP.someLocator} ${loc.EJP.anotherLocator}
NLO-L06 locator-variable-name-missing-parent-locator	When using multiple functional parts, they have to describe a parent-child relationship.	${loc.EJP.horizontalStepper.nextButton}
NLO-L07 locator-variable-name-camelcase	Functional parts must be camelCased.	${loc.EJP.someLocator}
NLO-L08 locator-variable-type	All locators must be written as XPath expressions.	//form//button
NLO-L09 locator-variable-with-meaningless-html-tag	HTML like and can be anything and nothing. Don’t use them as it makes your locators unstable, hard to read, more complex, and unspecific.	//div/div/form/div/span
NLO-L10 locator-variable-with-text	XPath supports the text() function. You can use this to locate an element by the text contents. However, doing this will make your tests harder to read.	//*[text() = 'Hello world']
NLO-L11 locator-comment-on-same-line	Always place the comment on its own line above the locator variable or locator variable group.	# Some comment ${loc.EJP.locator} ...
NLO-L012 locator-outside-variable	Each locator lives inside a variable. This allows us to easily reuse the same locator in multiple places. It also allows us to use a human-readable variable names which improves readability.	Click Element //button
NLO-T01 missing-code-in-test-name	Every test name must start with a test code. These test codes are used to distinguish between tests in pipeline run results and general communication.	GEN01-01 My Test
NLO-T02 inconsistent-test-code-prefix	Every test name in a file must start with the same test code prefix. It makes no sense to have multiple test code prefixes in one file.	GEN01-01 My Test EJP01-02 Another Test
NLO-T03 duplicate-test-code	Tests codes are unique identifiers for tests. Because of this, it makes no sense for the same test code to appear multiple times in one file.	GEN01-01 My Test GEN01-01 Another Test

Slides