Secure Access of Performance Monitoring Unit by User Space Profilers: White Paper [Revision 1.0 ed.]

Citation preview

Se cu r e Acce ss of Pe r for m a n ce M on it or in g Un it by Use r Spa ce Pr ofile r s W h it e Pa pe r This paper a proposes a soft ware m echanism t arget ing perform ance profilers which would run at user space privilege t o access perform ance m onit oring hardware, t he lat t er requires privileged access in kernel m ode, in a secure m anner wit hout causing unint ended int erference t o t he soft ware st ack.

Ju n e 2 0 1 6 Re vision 1 .0

Docum ent Num ber: 334467- 001

Not ice: This docum ent cont ains inform at ion on product s in t he design phase of developm ent . The inform at ion here is subj ect t o change wit hout not ice. Do not finalize a design wit h t his inform at ion. I nt el t echnologies’ feat ures and benefit s depend on syst em configurat ion and m ay require enabled hardware, soft ware, or service act ivat ion. Learn m ore at int el.com , or from t he OEM or ret ailer. No com put er syst em can be absolut ely secure. I nt el does not assum e any liabilit y for lost or st olen dat a or syst em s or any dam ages result ing from such losses. You m ay not use or facilit at e t he use of t his docum ent in connect ion wit h any infringem ent or ot her legal analysis concerning I nt el product s described herein. You agree t o grant I nt el a non- exclusive, royalt y- free license t o any pat ent claim t hereaft er draft ed which includes subj ect m at t er disclosed herein. No license ( express or im plied, by est oppel or ot herwise) t o any int ellect ual propert y right s is grant ed by t his docum ent . The product s described m ay cont ain design defect s or errors known as errat a which m ay cause t he product t o deviat e from published specificat ions. Current charact erized errat a are available on request . This docum ent cont ains inform at ion on product s, services and/ or processes in developm ent . All inform at ion provided here is subj ect t o change wit hout not ice. Cont act your I nt el represent at ive t o obt ain t he lat est I nt el product specificat ions and roadm aps. I nt el disclaim s all express and im plied warrant ies, including wit hout lim it at ion, t he im plied warrant ies of m erchant abilit y, fit ness for a part icular purpose, and non- infringem ent , as well as any warrant y arising from course of perform ance, course of dealing, or usage in t rade. Copies of docum ent s which have an order num ber and are referenced in t his docum ent m ay be obt ained by calling 1- 800- 548- 4725 or by visit ing www.int el.com / design/ lit erat ure.ht m . I nt el, t he I nt el logo, and Xeon are t radem arks of I nt el Corporat ion in t he U.S. and/ or ot her count ries. * Ot her nam es and brands m ay be claim ed as t he propert y of ot hers. Copyright © 2016, I nt el Corporat ion. All Right s Reserved.

2

Docum ent Num ber: 334467- 001, Revision 1.0

I nt roduct ion

Con t e n t s 1 

I nt r oduct ion ........................................................................................ 5   1.1  

2 

Scope ....................................................................................... 6  

I m ple m e nt a t ion .................................................................................. 7   2.1   2.2   2.3   2.4   2.5   2.6  

Securit y Model ........................................................................... Access Layer Requirem ent s .......................................................... Sharing Model ............................................................................ Archit ect ural Perfm on vs. Model Specific ........................................ Count er Wrapping ...................................................................... List of Regist ers for Secure Access by User- Space Profilers ...............

7  7  8  8  8  8 

Ta ble s Table 2- 1. Configurat ion Regist ers for PMU and Non- PMU Count ers ............. 9   Table 2- 2. PMU Count er Regist ers ........................................................... 9   Table 2- 3. Ot her Count er Regist ers1 ....................................................... 10  

Docum ent Num ber: 334467- 001, Revision 1.0

3

Re vision H ist or y

4

D ocu m e n t N u m be r

Re vision N u m be r

334467- 001

1.0

D e scr ipt ion  I nit ial release of t he docum ent .

Date June 2016

Docum ent Num ber: 334467- 001, Revision 1.0

I nt roduct ion

1

I n t r odu ct ion

Perform ance m onit oring unit s ( PMUs) are present in all m odern I nt el processor generat ions, allowing profiling ut ilit ies t o charact erize t he int eract ion bet ween soft ware and CPU resources using a rich set of perform ance m et rics. Profilers are crit ical t ools for soft ware t o harvest opt im al perform ance out of t he CPU hardware. The program m ing int erfaces t hat profiling ut ilit ies use t o access PMUs or relat ed hardware resources consist of: 





A set of inst ruct ions ( som e require privilege access available only in kernel m ode, like RDMSR, WRMSR) . PMU configurat ion resources: t hese are t ypically Model Specific Regist ers ( MSRs) . Count er regist er resources: t hese can include perform ance count ers in t he PMU as well as ot her count er regist ers accessible as MSRs.

Tradit ionally, profiling ut ilit ies em ploy special device drivers operat ing wit h ring 0 privilege t o configure t he PMU, access count er regist ers, and handle int errupt s if t he profiler support s sam pling ( i.e. capt ure sam ples of increm ent al dat a at fine- grain int ervals) . Som e OS, such as Linux, provide API access for root privileged user program s t o access privileged resources ( such as MSRs) . When a user program ’s profiling needs can be served by count ing of event s ( wit hout t he need t o capt ure increm ent al sam ples) , it is oft en possible and desirable t o im plem ent t he profiler as a ring 3 applicat ion t o m ake use of t hese privileged API s. This sim plifies developm ent and deploym ent of t he profiler com pared t o t he t radit ional approaches of a kernel based driver solut ion wit h a com m and line front - end parser. For securit y reasons in m ult i- user OS, t he OS only allows access t o privileged resources by root users. This im plies t hat t he m onit oring t ool would run wit h full root right s and have right s t o operat e privileged resources ( as perm it t ed by t hose API ) beyond j ust m onit oring perform ance event s. To configure and use t he PMU, read and writ e accesses t o som e PMU MSRs are needed by a user- space profiler. However, having full writ e access t o t he ent ire set of MSRs in a CPU can com prom ise t he OS. Thus, full root right s and writ e access t o full set MSRs should be select ively provisioned t o a user- space profiler. On secured shared server syst em s or securely boot ed client s wit h secured kernels full MSR access is usually not available. The goal of t his whit e paper is t o define a subset of MSRs and m echanism wit h t he following in m ind: 

Writ es t o t he subset of MSRs are t o configure perform ance m et ric select ion and conduct m onit oring of t he count er regist ers, wit hout changing any non- PMU st at es.

Docum ent Num ber: 334467- 001, Revision 1.0

5

I nt roduct ion

  

Define writ e m asks t hat are applicable t o t he subset of MSRs t o ensure t he userspace profiler operat es wit hin t he int ended m onit oring m ode ( i.e. count ing) . A bridge bet ween t he OS- API requirem ent of full root right s and t he desired nonroot perm ission for user- space applicat ions. Allow collect ing perform ance m et rics of t he whole syst em , but do not m odify any ot her st at e.

A specialized MSR access layer can t hen give t he m onit oring t ool only access t o t his safe “ m onit oring only” subset of MSRs and allow it t o run t he m onit oring as non- root , wit hout risking com prom ising t he syst em . Not e t hat m onit oring access is st ill opt - in by t he adm inist rat or and cannot be done wit hout an explicit configurat ion change.

1 .1

Scope

The scope is largely focused on m onit oring for t he processor core PMU. I nt el plat form s have addit ional PMUs out side t he processor core such as t he uncore or t he chipset . Those are not covered by t his whit e paper.

§

6

Docum ent Num ber: 334467- 001, Revision 1.0

I m plem ent at ion

2 2 .1

I m ple m e n t a t ion Se cu r it y M ode l

This whit e paper define a new “ global m onit oring only” privilege level for an applicat ion. The adm inist rat or has t o explicit ly grant t his privilege level t o an applicat ion. The privilege allows m onit oring perform ance event s on all processes of t he com plet e syst em , but does not change any global st at e not accessible by an unprivileged applicat ion. The privilege level gives read and writ e access t o a lim it ed num ber of MSRs in t he logical processor and t he physical package. Filt ering of input set t ings specified by t he applicat ion is writ t en t o t he MSRs by a privileged soft ware layer ( kernel driver or a special secure access layer) . The act ive set t ings of t he MSRs reflect t he configurat ion of t he perform ance m onit oring hardware. I nput from t he non- root applicat ion t o change any of t he secured m onit oring regist ers does not allow: 







Reading or writ ing any dat a in m em ory or in dat a regist ers. Triggering int errupt s. Changing st at e of processes out side t he m onit oring t ool. I n general, t he expect at ion of perform ance im pact t o t he t arget syst em due t o enabling m onit oring hardware and t he soft ware layer is m inim al.

I nput from t he non- root applicat ion perm it t he following changes t o t he secured m onit oring regist ers: 



Select ion of perform ance m onit oring count er event s which are support ed by t he PMU, as well as ( opt ionally) condit ioning of perform ance count er result s ( e.g. t hresholding, edge t riggering) . This includes t he abilit y t o m onit or event s such as cache m isses, branch m ispredict ions and ot her archit ect ural and m icro archit ect ural event s.

The adm inist rat or can choose whet her ring 0 ( kernel) or only all user m ode can be m onit ored.

2 .2

Acce ss La ye r Re qu ir e m e n t s

The secure access layer should im plem ent t he following funct ionalit y: 



Allow specific soft ware access wit hout requiring t he soft ware t o run wit h full adm inist rat or right s. Allowing access t o specific whit e list ed MSR regist ers, as docum ent ed in t his docum ent .

Docum ent Num ber: 334467- 001, Revision 1.0

7

I m plem ent at ion

 

Enforce t hat som e regist ers are read only and t hat som e regist ers have bit s writ e prot ect ed. Cat ching # GP General Prot ect ion fault s when accessing MSRs and ret urn an error.

2 .3

Sh a r ing M ode l

Writ e access t o t he PMU regist ers by one global m onit oring soft ware process can dist urb ot her m onit oring t ools operat ing under t he sam e syst em execut ive. To allow sharing bet ween different m onit oring t ools t he t ool should follow t he prot ocol specified in t he I nt el Perform ance Monit oring unit sharing guide ( www.int el.com / sdm or ht t ps: / / soft ware.int el.com / file/ 30388) . Generally t his m eans checking enable bit s for program m able count ers and not changing t he configurat ion if t he count er is already running. Free running count ers can be always shared, but should not be writ t en t o.

2 .4

Ar ch it e ct u r a l Pe r fm on vs. M ode l Spe cific

Som e regist ers are archit ect ural and can be discovered t hrough t he CPUI D inst ruct ion. Ot her regist ers are m odel specific.

2 .5

Count e r W r a pping

Wit h t he secure access rest rict ions it is not possible t o get an int errupt on count er overflow. Soft ware inst ead needs t o poll t he count er regist ers in sufficient ly short t im e int ervals t o accum ulat e values before t hey overflow.

2 .6

List of Re gist e r s for Se cu r e Acce ss by Use r Spa ce Pr ofile r s

MSR regist ers available in I nt el processors for user- space profilers via a secure access layer are list ed below. Availabilit y of a given MSR in an I nt el processor is enum erat ed eit her by CPUI D feat ure inform at ion or by m odel- specific signat ures report ed in Display_Fam ily, Display_Model values ret urned in CPUI D inst ruct ion leaf 1 funct ion. I n general, only I nt el processors wit h DisplayFam ily = 0x6 are applicable t arget s of t his paper. MSR inform at ion applicable t o DisplayModel values of 0x1E, 0x1F, 0x1A, 0x2F, 0x25, 0x2C, 0x2E, 0x37, 0x4D, 0x4C, 0x1C, 0x26, 0x27, 0x36, 0x35, 0x2A, 0x2D, 0x3A, 0x3E, 0x3C, 0x45, 0x46, 0x3C, 0x3F, 0x3D, 0x47, 0x56, 0x4E, 0x5E, 0x57 are sum m arized by cat egory. Unless ot herwise m arked all bit s in t he regist er can be securely accessed. N ot e :

8

For m ore det ails on t he individual regist ers, see t he I nt el ® 64 and I A- 32 Archit ect ures Soft ware Developer Manuals ( www.int el.com / sdm ) .

Docum ent Num ber: 334467- 001, Revision 1.0

I m plem ent at ion

Ta ble 2 - 1 . Configu r a t ion Re gist e r s for PM U a n d N on- PM U Coun t e r s Nam e

Acce ss

Addr e ss

D e scr ipt ion

Scope

Com m e n t s

I A32_PERF_EVENTSE Lx

R/ W

0x186+ x, x = 0, n- 1; n = CPUI D.10: EAX[ 15: 8]

Select perform ance m onit or ing event s and associat ed configurat ions.

Thread

Ring 0 access m ask 0xffa7ffff, ot herwise 0xffa5ffff

I A32_FI XED_CTR_CT RL

R/ W

0x38d

Configur e fixed count ers.

Thread

Ring 0 access m ask 0x333, ot herwise 0x111

I A32_PERF_GLOBAL_ CTRL

R/ W

0x38f

Global cont rol t o enable/ disable fixed count ers and perform ance count ers.

Thread

Access m ask 0xff000000ff

MSR_OFFCORE_RSP_ 0

R/ W

0x1a6

Configur e event - specific m ask for OFFCORE_RSP_0 event .

Varies

Writ ing reserv ed bit s m ay # GP;

MSR_OFFCORE_RSP_ 1

R/ W

0x1a7

Configur e event - specific m ask for OFFCORE_RSP_1 event .

Varies

Writ ing reserv ed bit s m ay # GP

I A32_PERF_CAPABI LI TI ES

R/ O

0x345

Enum erat e Per fm on capabilit y.

Thread

MSR_RAPL_POWER_ UNI T

R/ O

0x606

Enum erat e Granular it y of RAPL Energy St at us Count ers.

Package

Not available t o DisplayModels= 0x1E, 0x1F, 0x1A, 0x2E, 0x2F, 0x25, 0x2C, 0x1C, 0x26, 0x27, 0x35, 0x36

Ta ble 2 - 2 . PM U Cou n t e r Re gist e r s Nam e

Acce ss

M SR N u m be r

D e scr ipt ion

Scope

I A32_PERFCTRx

R/ W

0xc1+ x, x = 0, n- 1; n = CPUI D.10: EAX[ 15 : 8]

Value of count er x associat ed wit h configured perform ance event .

Thread

I A32_PMCx

R/ W

0x4c1+ x, x = 0, n- 1; n = CPUI D.10: EAX[ 15 : 8]

Full- widt h= writ able count er x.

Thread

I A32_FI XED_CTRx

R/ W

0x309+ x

Value of fixed count er x

Thread

Docum ent Num ber: 334467- 001, Revision 1.0

Com m e n t s

9

I m plem ent at ion

Ta ble 2 - 3 . Ot h e r Coun t e r Re gist e r s 1 Nam e2

Acce ss

Addr e ss

D e scr ipt ion

Scope

Com m e n t

MSR_PKG_Cx_RESI DENCY

R/ O

Varies by Available Cx

Durat ion in applicable package C st at es.

Package

See Chapt er 35 of t he I nt el ® 64 and I A- 32 Archit ect ures Soft ware Developer Manual, Volum e 3C ( www.int el.com / sdm )

MSR_CORE_C1_RESI DENCY

R/ O

0x660

Durat ion in core C1 st at e.

Core

Only in Display Models= 0x37, 0x4D, 0x4A, 0x5A, 0x5D, 0x4C, 0x5C, 0x5F

MSR_CORE_C3_RESI DENCY

R/ O

0x3fc

Durat ion in core C3 st at es.

Core

Not available t o DisplayModels= 0x1E, 0x1F, 0x1A, 0x2E, 0x2F, 0x25, 0x2C, 0x1C, 0x26, 0x27, 0x35, 0x36, 0x37, 0x4D, 0x4A, 0x5A, 0x5D

MSR_CORE_C6_RESI DENCY

R/ O

0x3fd

Durat ion in core C6 st at es.

Core

Not available t o DisplayModels= 0x1E, 0x1F, 0x1A, 0x2E, 0x2F, 0x25, 0x2C, 0x1C, 0x26, 0x27, 0x35, 0x36, 0x37, 0x4D, 0x4A, 0x5A, 0x5D, 0x4C, 0x5C, 0x5F

I A32_APERF

R/ O3

0xe8

Act ual perform ance clock count .

Thread

I A32_MPERF

R/ O3

0xe7

TSC clock count .

Thread

MSR_PPERF

R/ O

0x64e

Product iv e perform ance count .

Thread

Only on DisplayModels= 0x4E, 0x5E

MSR_PKG_ENERGY_STATUS

R/ O

0x611

RAPL energy of t he package.

Package

Not available t o DisplayModels= 0x1E, 0x1F, 0x1A, 0x2E, 0x2F, 0x25, 0x2C, 0x1C, 0x26, 0x27, 0x35, 0x36

MSR_SMI _COUNT

R/ O

0x34

Syst em m anagem ent int errupt count .

Thread

Not available t o DisplayModels= 0x1C, 0x26, 0x27, 0x35, 0x36

N OTES: 1. A m achine readable ver sion of t his t able can be downloaded from ht t ps: / / download.01.org/ perfm on/ secur e- pm u- access- 1.0.csv. 2. The availabilit y and address of som e MSRs list ed in t his t able m ay vary by Display Fam ily _Display Model signat ur es; refer t o Chapt er 35 of t he I nt el ® 64 and I A- 32 Archit ect ures Soft ware Developer Manual, Volum e 3C ( www.int el.com / sdm ) . 3. Recom m ended access layer t o enforce read- only for bet t er sharing, how ever allowing writ e access does not com prom ise secur it y.

10

Docum ent Num ber: 334467- 001, Revision 1.0