在Android中使用ARM Coresight ETM/ETE进行指令流分析的方法及平台选型
关于CoreSight
这篇文章不打算介绍CoreSight基本原理,本文假设读者已经对CoreSight的基本原理以及作用有了基本了解,打算使用CoreSight来做TA的分析或者App及内核的高效Trace和重放。本文主要给出一种获得容易获得的Coresight ETM/ETE设备平台的渠道。
对于Coresight相关的基本介绍,原理,可以参考这些文章:
coresight(四) channel interface
CoreSight在ARM TA安全研究中的应用论文:
Google的文档关于如何在Android中开启Coresight ETM以及ETM在AutoFDO中的应用:
Linux基金会在内核中使用MEM-AP整合Coresight ETM所做的工作:
https://www.slideshare.net/linaroorg/lcu14-101-coresight-overview
https://elinux.org/images/b/b3/Hardware_Assisted_Tracing_on_ARM.pdf
支持CoreSight的硬件选型
虽然理论上来说,只要是ARM平台,Coresight ETM宏单元是公版ARM核心的一部分,只要是ARM架构都可以使用,但实际情况并非如此,以下我们分硬件类别来说明:
服务器平台
目前常见的ARM服务器平台有:
华为鲲鹏服务器 华为鲲鹏服务器(以鲲鹏920为代表)明确支持Coresight ETM,可以在Linux中对程序进行动态Trace,但可惜的是,目前华为并不开放Coresight驱动,以至于外部用户无法使用。
Marvell ThunderX2服务器 明确支持Coresight ETM,也有相关的驱动和开源案例,但由于该平台很小众,已停产,买不着。
Ampere Computing服务器产品 以Ampere Altra和AmpereOne架构为代表,支持Coresight ETM,但目前根据笔者测试,虽然Ampere官方放出了了Coresight相关的驱动,但问题在于,目前各大厂商主板的UEFI固件并未将Coresight硬件通过ACPI协议报告给上层系统(可能因为觉得这个功能没什么人使用),因此目前仍然无法使用。
开发板
RockChip RK3399开发板 根据Rockchip官方的介绍,RK3399芯片是支持完整的Coresight ETM功能的(RK3588/3588S等后续产品阉割了这部分支持),但目前并没有开源的设备树启用它,因此暂时没有现成方案,如果要使用需要自己适配。
NVIDIA Jetson TX2开发板 明确支持,可用。
NVIDIA Jetson Nano开发板 明确支持,可用。
NVIDIA平台已有开源拿项目可以拿来用,参考:https://github.com/RICSecLab/coresight-trace
手机
根据笔者的探索,高通方案的手机一直支持Coresight ETM功能(从8gen1使用armv9架构后,ETM被同功能的ETE单元取代),但很遗憾,量产机型都对SoC开启了Secureboot(启用secureboot的过程被称之为熔断),也就是熔断,熔断之后的机器,其Debug功能会被关闭,如果要开启,需要通过熔断时配置的证书签名的Debug Policy来启用。但量产机型中,证书掌握在OEM厂商手中,一般不会泄漏出来,因此ETM功能处于不可用的状态。
对于Coresight的认证过程根据ARM官方文档的资料,CoreSight DEBUG和Trace模块接受4个个引脚的控制:DBGEN,SPIDEN,NIDEN,SPNIDEN。
但这几个信号很多时候并不是以引脚的形式留在外面,查阅了一些资料,其信号使能方式和SoC的具体设计有关系,因为ARM仅仅提供了Core的实现,而芯片要工作,往往需要加上外围的IO模块,因此各家厂商配用的模块不同,最终的设计可能不同。但由于各厂商对认证的方式公开资料较少,在这里只整理了一些已知的信息。
- 高通平台:当soc未配置secureboot时,debug权限默认开启,当secureboot开启后,debug就会被关闭,此时需要oem厂商签名的debug policy镜像文件来通过auth认证开启权限。
- 三星Exynos平台:系列使用SJTAG模块,challenge response方式认证,本质还是证书。
- 飞思卡尔iMX系列:这些信号接在DAP模块上,需要外部调试器操作DAP寄存器来启用这些信号。
- Google Silicon GS系列:认证方案和三星Exynos系列类似,推测GS系列芯片是三星Exynos的半定制产品
下面着重比较各类可用的平台
Pixel 系列工程机
目前能较方便获得的机型主要就只有Pixel 4/4 XL的EVT/DVT机型其他机型极少流出
以下是笔者测试的一台MP量产版Pixel 3a XL,即使在内核中编译了Coresight ETM,驱动,可见其AUTH寄存器仍然是0x88,也即debug功能(包括Secure world和non-Secure World当中的Trace功能)属于关闭状态,当Debug功能可用时,该寄存器的值应该为0xcc。
参考/sys/bus/coresight/devices/coresight-etm0/mgmt/trcauthstatus寄存器的值。(以下是量产版MP机型Pixel 3a XL的结果)
因此,如果要在高通平台的手机上快乐的使用ETM,需要满足两个条件:
- 非熔断机器:例如方案验证阶段的机型,比如EVT/DVT阶段的工程机(一旦进入PVT和MP阶段,就会熔断,此时已不可用),例如最方便的Pixel 4工程机(量大,目前仍可买到二手),高通原厂参考设计,例如QRD,MTP,HDK等机型。
- 与机型配套的源代码(至少需要有内核源代码以添加Coresight ETM设备树和驱动),这样的条件,Pixel由于Google给出源码,因此可以满足条件,而高通的QRD/MTP/HDK机型,如果有高通授权的QTI账号,或者特殊渠道,也可以获得相关源码。
对于Pixel 4的非熔断机器(EVT DVT),可以清楚的看见Secureboot为NONE,即没有开启
从系统中也可以清楚的看到,trcauthstatus的值为0xcc,即ETM为开启状态。
但可惜的是,目前Pixel系列的工程机,仅能买到n手Pixel 4或者Pixel 4 XL,且内核版本在4.9~4.14不等,而如果要结合ebpf分析,无法使用5.10以后版本内核中的新patch(安卓系统使用的高通内核无法升级到更高版本)。
高通QRD/MTP/HDK工程机
高通原厂工程机,可以获得最新的SoC,且没有熔断,而且内核源码一般都会开放(例如小米,oneplus或者OPPO一般都会在github上开源内核源码),可自行定制功能。有多种机型、平台可以选择,数量也可保证,如果你对高通工程机感兴趣,欢迎交流。
QRD888:
QRD8450(8 gen 1)
MTP8550(8 gen 2)
QRD/MTP/HDK机型均未熔断,Secureboot为关闭状态,此时ETM可用:
MTP8450(8 gen 1)
/sys/bus/coresight/devices/coresight-ete0/mgmt/trcauthstatus的值为0xcc可用
同时可以使用simpleperf进行trace或者inject:
taro:/data/local/tmp # simpleperf record -e cs-etm --duration 3 -a simpleperf W dso.cpp:434] failed to read symbols from [qcom_hwspinlock]: File not found simpleperf W dso.cpp:434] failed to read symbols from [smem]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_wdt_core]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gh_virt_wdt]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_ipc_logging]: File not found simpleperf W dso.cpp:434] failed to read symbols from [cmd_db]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_rpmh]: File not found simpleperf W dso.cpp:434] failed to read symbols from [proxy_consumer]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gdsc_regulator]: File not found simpleperf W dso.cpp:434] failed to read symbols from [icc_bcm_voter]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_ipcc]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_tsens]: File not found simpleperf W dso.cpp:434] failed to read symbols from [rpmh_regulator]: File not found simpleperf W dso.cpp:434] failed to read symbols from [dcvs_fp]: File not found simpleperf W dso.cpp:434] failed to read symbols from [icc_rpmh]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_dcvs]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_cpufreq_hw]: File not found simpleperf W dso.cpp:434] failed to read symbols from [thermal_pause]: File not found simpleperf W dso.cpp:434] failed to read symbols from [cfg80211]: File not found simpleperf W dso.cpp:434] failed to read symbols from [hwkm]: File not found simpleperf W dso.cpp:434] failed to read symbols from [crypto_qti_hwkm]: File not found simpleperf W dso.cpp:434] failed to read symbols from [dispcc_waipio]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gcc_diwali]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gh_arm_drv]: File not found simpleperf W dso.cpp:434] failed to read symbols from [iommu_logger]: File not found simpleperf W dso.cpp:434] failed to read symbols from [llcc_qcom]: File not found simpleperf W dso.cpp:434] failed to read symbols from [secure_buffer]: File not found simpleperf W dso.cpp:434] failed to read symbols from [mem_buf]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qti_fixed_regulator]: File not found simpleperf W dso.cpp:434] failed to read symbols from [reboot_mode]: File not found simpleperf W dso.cpp:434] failed to read symbols from [msm_geni_se]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gcc_waipio]: File not found simpleperf W dso.cpp:434] failed to read symbols from [phy_qcom_ufs]: File not found simpleperf W dso.cpp:434] failed to read symbols from [pinctrl_diwali]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_dload_mode]: File not found simpleperf W dso.cpp:434] failed to read symbols from [pmu_vendor]: File not found simpleperf W dso.cpp:434] failed to read symbols from [regmap_spmi]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_spmi_pmic]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qnoc_diwali]: File not found simpleperf W dso.cpp:434] failed to read symbols from [socinfo]: File not found simpleperf W dso.cpp:434] failed to read symbols from [stub_regulator]: File not found simpleperf W dso.cpp:434] failed to read symbols from [ufshcd_crypto_qti]: File not found simpleperf W dso.cpp:434] failed to read symbols from [mac80211]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_rimps]: File not found simpleperf W dso.cpp:434] failed to read symbols from [ufs_qcom]: File not found simpleperf W dso.cpp:434] failed to read symbols from [minidump]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gh_dbl]: File not found simpleperf W dso.cpp:434] failed to read symbols from [phy_qcom_ufs_qmp_v4_cape]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qnoc_qos]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_reboot_reason]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_llcc_pmu]: File not found simpleperf W dso.cpp:434] failed to read symbols from [clk_qcom]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gh_rm_drv]: File not found simpleperf W dso.cpp:434] failed to read symbols from [c1dcvs_scmi]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_cpu_vendor_hooks]: File not found simpleperf W dso.cpp:434] failed to read symbols from [sched_walt_debug]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qrtr]: File not found simpleperf W dso.cpp:434] failed to read symbols from [mem_offline]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_scm]: File not found simpleperf W dso.cpp:434] failed to read symbols from [phy_generic]: File not found simpleperf W dso.cpp:434] failed to read symbols from [phy_qcom_ufs_qmp_v4_diwali]: File not found simpleperf W dso.cpp:434] failed to read symbols from [msm_qmp]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gh_msgq]: File not found simpleperf W dso.cpp:434] failed to read symbols from [mem_buf_dev]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_pdc]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_dma_heaps]: File not found simpleperf W dso.cpp:434] failed to read symbols from [clk_rpmh]: File not found simpleperf W dso.cpp:434] failed to read symbols from [clk_dummy]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qti_regmap_debugfs]: File not found simpleperf W dso.cpp:434] failed to read symbols from [icc_debug]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_iommu_util]: File not found simpleperf W dso.cpp:434] failed to read symbols from [phy_qcom_ufs_qmp_v4_waipio]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_pmu_lib]: File not found simpleperf W dso.cpp:434] failed to read symbols from [msm_rtb]: File not found simpleperf W dso.cpp:434] failed to read symbols from [spmi_pmic_arb]: File not found simpleperf W dso.cpp:434] failed to read symbols from [rtc_pm8xxx]: File not found simpleperf W dso.cpp:434] failed to read symbols from [nfc_i2c]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_aoss]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qnoc_waipio]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qcom_gic_intr_routing]: File not found simpleperf W dso.cpp:434] failed to read symbols from [gh_ctrl]: File not found simpleperf W dso.cpp:434] failed to read symbols from [debug_regulator]: File not found simpleperf W dso.cpp:434] failed to read symbols from [cqhci]: File not found simpleperf W dso.cpp:434] failed to read symbols from [dispcc_diwali]: File not found simpleperf W dso.cpp:434] failed to read symbols from [pinctrl_msm]: File not found simpleperf W dso.cpp:434] failed to read symbols from [bwmon]: File not found simpleperf W dso.cpp:434] failed to read symbols from [pinctrl_cape]: File not found simpleperf W dso.cpp:434] failed to read symbols from [mem_hooks]: File not found simpleperf W dso.cpp:434] failed to read symbols from [sched_walt]: File not found simpleperf W dso.cpp:434] failed to read symbols from [nvmem_qcom_spmi_sdam]: File not found simpleperf W dso.cpp:434] failed to read symbols from [tmecom_intf]: File not found simpleperf W dso.cpp:434] failed to read symbols from [crypto_qti_common]: File not found simpleperf W dso.cpp:434] failed to read symbols from [arm_smmu]: File not found simpleperf W dso.cpp:434] failed to read symbols from [pmu_scmi]: File not found simpleperf W dso.cpp:434] failed to read symbols from [msm_dma_iommu_mapping]: File not found simpleperf W dso.cpp:434] failed to read symbols from [qca6490]: File not found simpleperf W dso.cpp:434] failed to read symbols from [msm_geni_serial]: File not found simpleperf W dso.cpp:434] failed to read symbols from [memory_dump_v2]: File not found simpleperf W dso.cpp:434] failed to read symbols from [cpu_hotplug]: File not found simpleperf W dso.cpp:434] failed to read symbols from [pinctrl_waipio]: File not found simpleperf W dso.cpp:434] failed to read symbols from [kryo_arm64_edac]: File not found simpleperf W dso.cpp:434] failed to read symbols from [c1dcvs_vendor]: File not found simpleperf W dso.cpp:434] failed to read symbols from [bcl_pmic5]: File not found simpleperf I cmd_record.cpp:750] Aux data traced: 58612288 taro:/data/local/tmp #
cs
大佬,想问下,有没有什么办法知道ARM SOC是否支持Coresight ETM?可以通过某些寄存器来查询吗?
Jarvis
一般都是支持的,取决于官方文档有没有告诉你寄存器的位置。以及有没有提供MEM-AP的驱动,一般单片机只提供了DAP