Introduction

Following part one, this is the second half of a two part tutorial series on how access a memory-mapped device implemented in Zynq’s programmable logic fabric.

Recap

So far we’ve built a new ZedBoard project from scratch. It has a pair-of-32-bit-counters peripheral in the programmable logic. Thanks to the XPS Base System Builder Wizard, its processing system is preconfigured with support for UART, GPIO, SD card, Quad SPI, USB, Ethernet, and 512 MB of DRAM. We’ve built an FSBL that sets this all up, loads the PL, and loads and launches u-boot; and we’re going to reuse the same good old u-boot.elf Linux boot loader from last time.

Notice the system we have just built from scratch does not include the nice ADI HDMI display controller. Certainly we could have added our counters to that system, but to focus on the essentials I thought a brand new, minimalist design would be better. Not to worry, we can still use desktop Ubuntu over the network from another computer.

Headless desktop Linaro Ubuntu Linux

This recapitulates headless operation from last time.

• Boot your Linaro Ubuntu system from SD card.
• Install the VNC and RDP servers. From serial port console or Terminal, $ sudo apt-get install xrdp . Then you will be able to boot and run headless. You may still need to use the serial port console to determine the DHCP-assigned IP address ($ ifconfig).
• On Windows 7, run Remote Desktop Connection (a.k.a. mstsc.exe). Specify your ZedBoard’s IP address, and log in. Voila, desktop Ubuntu over the network. HDMI monitor not required.

Now let’s build a devicetree.dtb that does not probe for not configure the HDMI display controller or any other PL peripherals.

• $ cd linux/arch/arm/boot/dts; cp zynq-zed-adv7511.dts zynq-zed-no-pl.dts
• Edit it to delete all programmable logic peripherals. It will look like this:

/dts-v1/;

/include/ "zynq-zed.dtsi"

• Back up your SD card.
• Build your new empty PL devicetree blob: $ cd linux; make zynq-zed-no-pl.dtb
• Copy it to the SD card. cp arch/arm/boot/zynq-zed-no-pl.dtb /media/BOOT/devicetree.dtb
• Boot Linaro Ubunto Linux on your ZedBoard. Notice your HDMI monitor is now blank!
• Fear not. On your TeraTerm serial port console, get the IP address ($ ifconfig), and connect and login to your ZedBoard via RDP or VNC. All should work as before, even though you are not using any of the programmable logic fabric.

Device access via /dev/mem

The first, simplest, most elemental, most hacky, most dangerous way to access our peripheral from Linux is by opening /dev/mem and using mmap() to map a view of the device’s physical address space into our process’s virtual address space. Sven Andersson’s UIO blog entry summarizes these considerations perfectly so I’ll just quote him extensively here:

“… Here are some of the characteristics:

• Userspace interface to system address space
• Accessed via mmap() system call
• Must be root or have appropriate permissions
• Quite a blunt tool – must be used carefully
• Can bypass protections provided by the MMU Possible to corrupt kernel, device or other processes memory

Pro

• Very simple – no kernel module or code
• Good for quick prototyping / IP verification
• peek/poke utilities
• Portable (in a very basic sense)

Con

• No interrupt handling possible
• No protection against simultaneous access
• Need to know physical address of IP Hard-code?”

Continuing with our zed_counters project, with headless Ubuntu set up, let’s reboot our Linux system, configured to use the zed_counters’ design BOOT.BIN image we built earlier.

• Copy it to your SD card $ cp …/zed_fsbl/bootimage/u-boot.bin /media/BOOT/BOOT.BIN

The SD card BOOT partition now contains our new BOOT.BIN with the zed_counters design; a new devicetree.dtb blog which denies any devices in the PL fabric, and the uImage Linux kernel we built last time. Safe-eject it, insert it into your ZedBoard, and reboot. Start a remote desktop connection. Open a Terminal. Fetch this /dev/mem access-based test application from Andersson’s blog entry.

• $ wget http://svenand.blogdrive.com/files/gpio-dev-mem-test.c
• Read the code. It opens /dev/mem read/write, mmaps one page, and loads or stores through that virtual address.

	/* Open /dev/mem file */
	fd = open ("/dev/mem", O_RDWR);
	if (fd < 1) {
		perror(argv[0]);
		return -1;
	}

	/* mmap the device into memory */
	page_addr = (gpio_addr & (~(page_size-1)));
	page_offset = gpio_addr - page_addr;
	ptr = mmap(NULL, page_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, page_addr);

• There’s nothing GPIO specific about it, so it can be used to test our counters device.
• Build it. $ cc -o gpio-dev-mem-test gpio-dev-mem-test.c
• Run it. $ ./gpio-dev-mem-test -g 0x70000000 -i =>  “GPIO access through /dev/mem. ./gpio-dev-mem-test: Permission denied”
• Oops! This highlights a shortcoming of the /dev/mem approach. /dev/mem is a character special file owned by root and unreadable by regular users:
  $ ls -l /dev/mem => crw-r—– 1 root kmem 1, 1 Dec 31 1969 /dev/mem
• Make the test program SETUID root so it can open the file. $ sudo chown root gpio-dev-mem-test; sudo chmod u+s gpio-dev-mem-test Congratulations, you have now added a massive security and robustness hole to your Linux system.
• Try again. Read (write) the up-counter at address 0x70000000:
• $ ./gpio-dev-mem-test -g 0x70000000 -i => … input: 00000000
• $ ./gpio-dev-mem-test -g 0x70000000 -i => … input: 00000001
• $ ./gpio-dev-mem-test -g 0x70000000 -i => … input: 00000002
• $ ./gpio-dev-mem-test -g 0x70000000 -o 7
• $ ./gpio-dev-mem-test -g 0x70000000 -i => … input: 00000007
• $ ./gpio-dev-mem-test -g 0x70000000 -i => … input: 00000008
• It counts! Read the free-running counter at address 0x70000004:
• $ ./gpio-dev-mem-test -g 0x70000004 -i => … input: 577c9e79
• $ ./gpio-dev-mem-test -g 0x70000004 -i => … input: 62502c56
• $ ./gpio-dev-mem-test -g 0x70000004 -i => … input: 6b764783
• It counts at 100 MHz.

It’s great that we can kick the tires on our new device without any device drivers or kernel configuration, but as Mr. Andersson writes, direct /dev/mem access is “OK for prototyping – not recommended for production.” Next we’ll see how to employ user-mode I/O to access just the counters device — with better safety and security.

Device Access via UIO

In this final section, we will see how to configure, modify, and rebuild your Linux kernel so you can configure UIO devices in your device tree, and how to access them from your application program.

Your ADI or Xilinx Linux source tree already contains the source to two useful UIO drivers, uio_pdrv (“UIO platform driver”) and uio_pdrv_genirq (“UIO platform driver with generic interrupts”). Unfortunately the default ADI and Xilinx build configurations do not build or include these drivers in the kernel (nor as loadable driver modules).

To enable these drivers run $ cd linux; make menuconfig. This pops a character-mode GUI titled .config — … Kernel Configuration.

• In Kernel Configuration, select Device Drivers —>
• In Device Drivers, select Userspace I/O drivers —>
• In Userspace I/O drivers, select <*> as built-in both Userspace I/O platform driver and Userspace I/O platform driver with generic IRQ handling.
• Exit. Exit. Exit. Save. Check: $ grep UIO .config =>
  CONFIG_UIO=y
  CONFIG_UIO_PDRV=y
  CONFIG_UIO_PDRV_GENIRQ=y

Now (before we rebuild the kernel) it is time to briefly review the UIO drivers kernel source. This won’t hurt much.

• $ cd linux/drivers/uio
• Review uio.c, uio_pdrv.c, and uio_pdrv_genirq.c.
• Note that uio.c contains common support routines that are used by the uio_pdrv and uio_pdrv_genirq drivers.
• Note that only uio_pdrv_genirq defines a devicetree/”open firmware device id” compatible key:  … { .compatible = “generic-uio”, },
• Reviewing the driver change history in github, we see this line was added in 11/2012 for “microblaze: UIO setup compatible property” to “Setup compatible property which matches petalinux”.
• So “out of the box”, after enabling UIO drivers in menuconfig, the only UIO driver you can use that will properly initialize with a device tree blob configuration is uio_pdrv_genirq, and to use that, you must describe your peripheral as a “generic-uio”.
• Furthermore, the uio_pdrv_genirq requires the device tree blob to also define the device interrupt number and the interrupt parent device.
• To make this work for our interrupt-less counters device, we can lie, pick a free interrupt number, and pretend our counters are wired up to the Zynq GIC interrupt controller, just like interrupt-issuing Zynq peripherals do. Interrupt numbers are biased by -32 for some reason. Here is what the ensuing DTS device tree specification looks like:

/dts-v1/;

/include/ "zynq-zed.dtsi"

/ {
    counters@70000000 {
        compatible = "generic-uio";
        reg = < 0x70000000 0x1000 >;
        interrupts = < 0 57 0 >;
        interrupt-parent = <&gic>;
    };
};

• We declare this device is a “generic-uio”, which will match uio_pdrv_genirq;
• Its registers are at physical address 0x70000000 and are 0x1000 (4 KB) in size.
• It generates interrupt 57+32 = 89. These are issued to the GIC. Not really.
• Where did the fake interrupt number 57+32=89 come from? I reviewed the interrupt assignments in various DTS files that may intersect this project, such as arch/arm/boot/dts/{zynq.dtsi, zynq-zed.dtsi, zynq-zed-adv7511.dts}, and picked one that wasn’t in use there.

Now we can (and I have) built a devicetree blob from this DTS file, booted Linux, and my device has probed, configured, initialized, and set up a /dev/uio0 (major device 251, minor device 0), and (as we’ll see below) I have accessed this from a user application.

In fact if we already had a peripheral with both memory-mapped I/O and interrupts, this existing driver uio_gen_pdrv would be ideal, we’d be done, and this seemingly endless tutorial would be over already.

But since example device counters doesn’t have interrupts, I am still not satisfied. What does it take to get the interrupt-less uio_pdrv working with device tree configuration?

• We have to modify drivers/uio/uio_pdrv.c to add an open firmware device id compatible string for it to match a device specification in the device tree. We’ll use the unremarkable name “uio_pdrv”.
• We also have to bring some code forward (and now, since I’m not a Linux kernel developer, this is getting dangerously close to cargo cult programming) into the uio_pdrv_probe() function to dynamically allocate a uio_info structure for the dynamically discovered uio_pdrv instance.
• For symmetry, we’ll add the compatible name “uio_pdrv_genirq” to the uio_pdrv_genirq device; it may then be used interchangeably with the existing name “generic-uio”.
• Here are the context diffs:

diff --git a/drivers/uio/uio_pdrv.c b/drivers/uio/uio_pdrv.c
index 72d3646..cc50312 100644
--- a/drivers/uio/uio_pdrv.c
+++ b/drivers/uio/uio_pdrv.c
@@ -14,6 +14,9 @@
 #include <linux/module.h> 
 #include <linux/slab.h>

+#include <linux/of.h>
+#include <linux/of_platform.h>
+
 #define DRIVER_NAME "uio_pdrv"

 struct uio_platdata {
@@ -28,6 +31,19 @@ static int uio_pdrv_probe(struct platform_device *pdev)
    int ret = -ENODEV;
    int i;

+   if (!uioinfo) {
+       /* devicetree -- alloc uioinfo for one device */
+       uioinfo = kzalloc(sizeof(*uioinfo), GFP_KERNEL);
+       if (!uioinfo) {
+           ret = -ENOMEM;
+           dev_err(&pdev->dev, "unable to kmallocn");
+           goto err_uioinfo;
+       }
+       uioinfo->name = pdev->dev.of_node->name;
+       uioinfo->version = "devicetree";
+       uioinfo->irq = UIO_IRQ_NONE;
+   }
+
    if (!uioinfo || !uioinfo->name || !uioinfo->version) {
        dev_dbg(&pdev->dev, "%s: err_uioinfon", __func__);
        goto err_uioinfo;
@@ -95,12 +111,23 @@ static int uio_pdrv_remove(struct platform_device *pdev)
    return 0;
 }

+#ifdef CONFIG_OF
+static const struct of_device_id uio_pdrv_of_match[] = {
+   { .compatible = "uio_pdrv", },
+   { },
+};
+MODULE_DEVICE_TABLE(of, uio_pdrv_of_match);
+#else
+# define uio_pdrv_of_match NULL
+#endif
+
 static struct platform_driver uio_pdrv = {
    .probe = uio_pdrv_probe,
    .remove = uio_pdrv_remove,
    .driver = {
        .name = DRIVER_NAME,
        .owner = THIS_MODULE,
+       .of_match_table = uio_pdrv_of_match,
    },
 };

diff --git a/drivers/uio/uio_pdrv_genirq.c b/drivers/uio/uio_pdrv_genirq.c
index 1f5ec28..3c7d8de 100644
--- a/drivers/uio/uio_pdrv_genirq.c
+++ b/drivers/uio/uio_pdrv_genirq.c
@@ -264,7 +264,8 @@ static const struct dev_pm_ops uio_pdrv_genirq_dev_pm_ops = {
 #ifdef CONFIG_OF
 static const struct of_device_id uio_of_genirq_match[] = {
    { .compatible = "generic-uio", },
-   { /* empty for now */ },
+   { .compatible = "uio_pdrv_genirq", },
+   { },
 };
 MODULE_DEVICE_TABLE(of, uio_of_genirq_match);
 #else

Now that we have configured UIO and added devicetree support to uio_pdrv, we are ready to rebuild the kernel and copy it to the SD card.

• $ cd linux; make uImage LOADADDR=0x00008000
• $ cp arch/arm/boot/uImage /media/BOOT

We can now use a nice simple devicetree .DTS for our zed_counters project:

/dts-v1/;

/include/ "zynq-zed.dtsi"

/ {
    counters@70000000 {
        compatible = "uio_pdrv";
        reg = < 0x70000000 0x1000 >;
    };
};

• Counters is now a uio_pdrv device with physical addresses 0x70000000-0x70000fff.
• No interrupt fakery required!
• Put that in arch/arm/boot/dts/zynq-zed-counters.dts, rebuild the device tree blob, and copy it to the SD card. $ make zynq-zed-counters.dtb;
  $ cp arch/arm/boot/zynq-zed-counters.dtb /media/BOOT/devicetree.dtb
• Together with the BOOT.BIN for our zed-counters design, we’re all set.
• Safe-eject the SD card, insert it into the ZedBoard, and boot Linux!
• There is no helpful console message confirming our counters device is initialized. That’s OK, we can see for ourselves.
• $ ls -l /dev/uio0 => crw——- 1 root root 251, 0 Dec 31  1969 /dev/uio0
• $ grep 251 /proc/devices => 251 uio
• $ cd /sys/devices/70000000.counters/uio/uio0; cat name version uevent =>
  counters
  devicetree
  MAJOR=251
  MINOR=0
  DEVNAME=uio0

Now to test UIO access to our counters. Andersson’s GPIO UIO test application is worthy of your study, but is not ideal to exercise our two counters. Instead, below, I have modified it (quickly hacked it) into a trivial counters-specific test. First we check counters[0] increments on every read. Next we do a few timing tests with our free running counters[1].

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
#include <sys/mman.h>
#include <fcntl.h>

#define MAP_SIZE 0x1000

void usage(void)
{
    printf("usage: test_counters -d <UIO_DEV_FILE>n");
}

int main(int argc, char *argv[])
{
    int c;
    int i;
    int fd = 0;
    char *uiod = 0;
    int value = 0;
    unsigned start, end;

    volatile unsigned *counters;

    while ((c = getopt(argc, argv, "d:io:h")) != -1) {
        switch(c) {
        case 'd':
            uiod = optarg;
            break;
        case 'h':
            usage();
            return 0;
        default:
            printf("invalid option: %cn", c);
            usage();
            return -1;
        }
    }

    if (!uiod) {
        usage();
        return -1;
    }

    /* Open the UIO device file */
    fd = open(uiod, O_RDWR);
    if (fd < 1) {
        perror(argv[0]);
        printf("Invalid UIO device file: '%s'n", uiod);
        return -1;
    }

    /* mmap the UIO device */
    counters = (volatile unsigned *)mmap(NULL, MAP_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
    if (!counters) {
        perror(argv[0]);
        printf("mmapn");
        return -1;
    }

    counters[0] = 1;
    assert(counters[0] == 1);
    assert(counters[0] == 2);
    assert(counters[0] == 3);

    counters[0] = 100;
    assert(counters[0] == 100);
    assert(counters[0] == 101);
    assert(counters[0] == 102);

    printf("wr-rd delay on free-running counter: ");
    for (i = 0; i < 20; ++i) {
        counters[1] = start = 0;
        end = counters[1];
        printf("%u ", end - start);
    }
    printf("n");

    printf("rd-rd delay on free-running counter: ");
    for (i = 0; i < 20; ++i) {
        counters[1] = 0;
        start = counters[1];
        end = counters[1];
        printf("%u ", end - start);
    }
    printf("n");

    munmap((void*)counters, MAP_SIZE);
    return 0;
}

Now let’s run it!

• $ cc -o counters counters.c
• $ ./counters -d /dev/uio0 => ./counters: Permission denied
• $ ls -l /dev/uio0 => crw——- 1 root root 251, 0 Dec 31  1969 /dev/uio0 Oh, right. The uio_pdrv driver created the device file but made it inaccessible to non-superusers.
• Fix that. $ sudo chmod 666 /dev/uio0
• $ ./counters -d /dev/uio0 =>
  wr-rd delay on free-running counter: 13 14 13 14 13 14 14 13 13 13 14 14 13 14 13 14 14 …
  rd-rd delay on free-running counter: 14 14 14 15 14 14 15 14 15 15 14 14 14 15 14 14 15 …

It works! Here we see that counter[0] correctly increments on every read, and successive read accesses to the free-running counter[1] each take 13-15 cycles e.g. 130-150 ns.

Unfortunately each time I reboot, permissions on /dev/uio0 revert to -rw——. I don’t know how to set them from within the device driver. Any suggestions? Thank you.

This ends the tutorial. Thank you for reading along. I hope it is of help in your Zynq Linux work. Please leave me your comments here or via twitter @jangray. Good luck and happy hacking.