For the second time in its history, cocotb has asked its users for input: how they are using cocotb, what they enjoy about it, what pain points they experience, and much more. The results are now in, and paint an encouraging picture. Cocotb not only works, but is enjoyed by many users, and the development priorities of the core development team match the expectations of our users.
The survey also identified some (not overly surprising) areas for improvement; primarily, the availability of learning resources and verification IP.
Read on for a more detailed look into the survey results. We’ll not dive too deep into ways to address the pain points of our users – there are some ideas, and a number of limitations to the abilities of a volunteer-driven project. Please reach out if you would like to get involved or have an idea!
About our users
Cocotb is typically used in industry by a small team of up to ten people.
Almost 60 percent of users use cocotb primarily in industry. That’s not surprising, given the challenges for hobbyists to design their own ASIC or FPGA design.
However, many users don’t stop using cocotb at work: when given the choice to select all their uses, the use in industry still leads with 65 percent, but is closely followed by the use in hobby projects (58 percent) and in academia (40 percent).
This first set of numbers already indicates: cocotb is a tool our users wish to use, not something they have to use (grumpily).
If you’re in industry, it’s rare that you’re doing verification on your own, and the answers reflect that. 46 percent of respondents work in a small team on a cocotb project, with 10 percent of our users even having multiple teams using cocotb!
Silicon design projects are often unique, and so are the projects our respondents use cocotb for. Common answers include:
- Image and video processing (computer vision)
- Packet processing
- Signal processing algorithms
- Processor verification (RISC-V)
- Full system testing/integration testing
What are our users verifying?
The typical cocotb user verifies an FPGA design written in Verilog.
Unsurprisingly, most cocotb users verify FPGA designs; with Verilog being the most popular language, not only among cocotb users.
What’s interesting is the increase in ASIC designs we’re seeing verified with cocotb: in 2020 (the last time we did the user survey), only 26 percent of our users were working on ASIC designs, compared to 42 percent in 2023, a growth of around 60 percent. With ASIC designers being traditionally more risk-averse (for good reasons), this increase can be seen as indication that cocotb is being trusted to deliver good verification outcomes.
Why not something else?
SystemVerilog UVM remains the primary verification choice – but cocotb users don’t really like it.
Cocotb isn’t the only verification approach out there, and (we have to admit) not the most common one either. SystemVerilog UVM has been the go-to choice for a while, and that’s reflected in our user base as well. Roughly one third of users use formal verification, plain VHDL testbenches, or other Python-based verification approaches. C or C++-based verification approaches (often combined with UVM) are used by 23 percent of our respondents.
The answers leave a lot of room for follow-up questions, like “how much of a design is actually verified using formal?” or “what does the in-house Python testbench look like?”. We’d certainly be interested in an answer – reach out over social media to tell us more!
When asked about the pain points of these other verification approaches, cocotb users repeatedly mentioned:
- the complexity (and learning curve) of UVM
- poor CI/test framework integration
- the amount of boilerplate code and effort required until the actual writing of the test can start
How our users run cocotb
The overwhelming majority of cocotb users uses a recent version of cocotb on Linux.
Cocotb is open source, and freedom of choice and diversity have long been key values in open source. Still, knowing the platforms the majority of our users run on is essential to properly focus our engineering and testing efforts. Thankfully, the results we got match what we were expecting.
The overwhelming majority of our users runs Linux, typically Red Hat Enterprise Linux or one of its derivatives, which tends to be most common in industry settings, or Debian or Ubuntu, which is used widely in “less managed” settings, as well as in CI.
Even though the cocotb project has spent a very significant amount of effort in engineering and support on it, our Windows user base remains comparably small.
Cocotb aims to support as many Python versions as possible, including versions which have been marked as “end-of-life” by the Python project itself (such as Python 3.6). Each new Python version brings significant benefits, with the most recent versions also being substantially faster in many use cases. We are therefore happy to see many users relying on recent or very recent versions of Python!
Many things can go wrong when one tries to compile Python by itself, as opposed to using a (RPM or deb) package prepared by a Linux distribution, conda, or python.org. We are therefore equally happy to see that the large majority of users makes use of these packages.
Cocotb pushes a new release roughly twice a year, and takes great care in ensuring backwards-compatibility to allow our users to switch to the latest version. The survey results confirm that this effort is paying off: 83 percent of users are on the latest 1.7 release (at the time of the survey).
A look at the simulators used with cocotb completes the picture. When choosing proprietary simulators, Cadence Xcelium and Siemens ModelSim and Questa lead the field. In the world of open source simulators, Verilator and Icarus Verilog are the most popular choices. Icarus Verilog is the default simulator in many cocotb tutorials and even its Makefiles, so its popularity is hardly surprising. Equally, GHDL is the most mature open source VHDL simulator, hence being the go-to choice for VHDL developers.
Finally, when asked for their wishlist of platforms cocotb should support, the most common answers were
- Support for the Vivado simulator (xsim) [which unfortunately does not implement the VPI or VHPI interface], and
- Better support for GHDL [which is partially due to the fact that GHDL is uniquely using the VPI interface for VHDL, as well as missing features in GHDL itself].
Verification IP or cocotb extensions
Most cocotb users use cocotb-test and a coverage extension, such as pyvsc or cocotb-coverage. Python implementations of UVM are popular with cocotb users as well, as is (AXI) bus IP.
Cocotb is intentionally not solving every possible problem: it’s extensible and encourages users to choose their preferred extensions from the Python package ecosystem.
For users who need more than cocotb’s Makefile-based build system, cocotb-test is the go-to choice. Functional coverage and constrained-random test generation are next on the list, with users mostly relying on pyvsc or cocotb-coverage.
As we’ll see below, the availability of verification IP (VIP) is one of the weak spots of cocotb. Still, high-quality VIP is available, and many users resort to cocotbext-axi for AXI bus adapters or cocotbext-spi for SPI adapters. Additionally, cocotb-bus provides a further set of bus interfaces which is commonly used.
Finally, despite many users choosing cocotb because it’s not UVM, a significant percentage of the respondents choose a UVM implementation in Python, with pyuvm being more popular than uvm-python, followed by in-house UVM-like methodologies.
Are users happy with cocotb?
Users love cocotb!
Cocotb users overwhelmingly love using it, and would recommend it to others. That’s enouraging to hear, and indicates that we’re on the right path. We are still not at the point where every cocotb user is fully satisfied (to the extent that’s ever possible), but overall the answers are very encouraging.
Looking at the more detailed answers, we see common themes emerge:
Users love cocotb because
- It is Python
- It is open source
- It integrates into the Python ecosystem
Users are least happy about
- The (lack of) tutorials and learning resources
- The (lack of) documentation
- The Makefile-based testrunner
- The availability of verification IP
Interestingly, roughly an equal amount of users love and hate the cocotb installation through pip install cocotb.
We’d love to hear more from those users who struggle with the process, since we invested significantly into this area in recent releases.
The same themes also reflect in reasons why cocotb isn’t used more in projects. 41 percent of users are looking for more verification IP, and 33 percent for commercial support, and 26 percent for commercial training. Finally, 31 percent are unsure about the maturity and future of cocotb.
If you are one of the users who are unsure about the future of cocotb: please reach out and let us know what you’re concerned about! Cocotb has just celebrated its 10th birthday and sees regular updates from an active user base – what are we missing?
A look at the cocotb (development) community
The cocotb community is welcoming and attracting new users, most of which are happy with cocotb as it is.
Most respondents to the survey are relatively new to cocotb, with around two thirds of users using cocotb for less than a year. We could see that as indication for a growing community!
This survey reached many of the most active cocotb users, and the answers on the interaction with the community reflect that as well. Still, 54 percent of our respondents have never opened a cocotb issue, and 75 percent have never contributed code through a pull request.
It is encouraging to see that most respondents see their interaction with the cocotb project as helpful, welcoming and fast. However, the picture isn’t as positive as it could be: 19 percent of respondents are unhappy with slow responses, 10 percent have experienced unproductive discussions. While it is important to stress that cocotb is a volunteer-driven project, there are certainly areas where we can improve our processes. These discussions are ongoing – please reach out if you have specific suggestions to make sure everybody feels supported and welcome in the best possible way!
About the survey itself
The cocotb user survey 2023 was open from May 30 to June 14, 2023 and attracted 48 answers. The survey was promoted over the cocotb blog, Twitter, LinkedIn, and the cocotb chat. We assume that this caused a bias towards the more active cocotb user base.
For comparision, the user survey 2020 did attract 111 responses. It is unclear where the difference in responses is coming from. One possible explanation might be the form of advertising chosen: The survey in 2020 was advertised widely in a banner on the cocotb documentation, something we did not do this year. All other statistics, such as PyPi download statistics, indicate a growing user base.