ASN.1 tools for Python

Whenever data structures are described in some machine and programming language independent and unambiguous way, such specification is called abstract syntax, by contrast with machine/language specific methods, which are called 'concrete' or 'transfer' syntaxes.

Abstract syntaxes appear useful in networking as a tool for engineering protocols in a clear and portable way. Moreover, once a protocol is described in some abstract language, protocol parsers and builders could be automatically generated for various computing architectures/programming languages, thus saving engineers from implementing low-level transport details by hand.

Abstract Syntax Notation One ( ASN.1 ) is a set of ITU standards defining particular implementation of abstract data description language accompanied by a collection of transfer encoding methods. Perhaps the most widely used among these data serialization methods is Basic Encoding Rules ( BER ) together with its derivatives ( DER and CER ), while Packed Encoding Rules PER ) aims at most compact data representation whilst in the wire.

This project is dedicated to implementation of ASN.1 types (concrete syntax) and codecs (transfer syntaxes) for Python programming environment. ASN.1 compiler is planned for implementation in the future.

Data model for ASN.1 types

The ASN.1 standard defines a set of primitive, scalar data types (such as Integer, String etc) and a few constructed types, each holding one or many other ASN.1 types as its components (constructed types may be viewed as Pascal "records" or C "struct"ures).

In pyasn1, those primitive ASN.1 types are implemented as immutable scalar objects. They could be used just like corresponding native Python types (integers, strings etc).

>>> from pyasn1.type import univ
>>> univ.Integer(12) - 2
10
>>> univ.OctetString('abc') == 'abc'
True
>>>

In ASN.1, constructed types (Sequence, SequenceOf, Set, SetOf, Choice) differ from each other by allowed components combination and ordering, which also projects to component addressing methods.

In this Python implementation, constructed ASN.1 types behave like Python sequence, and also support additional component addressing methods, specific to particular constructed type.

Components of Sequences can be addressed by their position in sequence:

>>> from pyasn1.type import univ, namedtype
>>> seq = univ.Sequence(componentType=namedtype.NamedTypes(namedtype.NamedType('version', univ.Integer())))
>>> seq.setComponentByPosition(0, univ.Integer())
>>> seq.getComponentByPosition(0)
Integer(0)
>>>

and by [textual] component type name (also valid for Set and Choice),

>>> seq.getComponentByName('version')
Integer(0)
>>>

as well as by type for Set and Choice:

>>> set = univ.Set(componentType=namedtype.NamedTypes(namedtype.NamedType('version', univ.Integer())))
>>> set.setComponentByPosition(0, univ.Integer())
>>> set.getComponentByType(univ.Integer().getTagSet())
Integer(0)
>>>

ASN.1 types are identified by a numeric ID called tag. In pyasn1, tags are implemented as immutable objects referred by ASN.1 type objects:

from pyasn1.type import tag
>>> tag.Tag(tag.tagClassUniversal, tag.tagFormatSimple, 3)
Tag(tagClass=0, tagFormat=0, tagId=3)
>>>

For the purpose of making same-typed objects distinguishable from one another, the standard allows for assigning custom tags to ASN.1 types. These tagged types preserve all properties of their parent type but possess different IDs.

There are two methods of tagging: implicit and explicit. The first one replaces base tag with arbitrary custom tag thus dropping all previously existing tag information for type:

>>> t = tag.TagSet(tag.Tag(tag.tagClassUniversal, tag.tagFormatSimple, 3))
>>> t.tagImplicitly(tag.Tag(tag.tagClassPrivate, tag.tagFormatSimple, 32))
TagSet(Tag(tagClass=192, tagFormat=0, tagId=32))
>>>

The explicit tag is build by appending new custom tag to already existing set of type's tags. Important property of explicit tagging is that it preserves base type information.

>>> t = tag.TagSet(tag.Tag(tag.tagClassUniversal, tag.tagFormatSimple, 3)
>>> t.tagExplicitly(tag.Tag(tag.tagClassPrivate, tag.tagFormatSimple, 32))
TagSet(Tag(tagClass=192, tagFormat=32, tagId=32), Tag(tagClass=0, tagFormat=0, tagId=3))
>>>

Besides tags, certain restrictions may be put upon ASN.1 types' values thus creating subtypes from base types (in computer science, a data type is a name of a collection of possible values). These restrictions are called subtype constraints in the ASN.1 standard.

Several different flavors of constraints exist. Some obvious include ValueRangeConstraint, ValueSizeConstraint and others. In pyasn1, constraints take shape of immutable objects capable of evaluating given value against constraint's specific logic.

>>> from pyasn1.type import constraint
>>> constraint.ValueRangeConstraint(1,2)
ValueRangeConstraint(1,2)
>>>

Multiple constraints can be combined altogether into sets with three basic boolean operations (ConstraintsUnion, ConstraintsIntersection and ConstraintsExclusion), which could be applied recursively.

>>> c = constraint.ConstraintsUnion(constraint.SingleValueConstraint(4), constraint.ValueRangeConstraint(-1, 2))
>>> c(1)
>>> c(3)
pyasn1.type.error.ValueConstraintError: ConstraintsUnion(SingleValueConstraint(4), ValueRangeConstraint(-1, 2)) failed at: all of (SingleValueConstraint(4), ValueRangeConstraint(-1, 2)) failed for 5
>>>

A constrainted ASN.1 type would then hold a reference to a top-most constraint object in a set and pass it a value, being assigned, for verification.

By evaluating the inclusion of all tags and constraints of one type in tag and constraint sets of another, it's possible to figure out the relationships between types. By way of background, types matching is used in constructed types for by-type component addressing.

>>> i1 = univ.Integer(subtypeSpec=constraint.SingleValueConstraint(0,3))
>>> i2 = univ.Integer(subtypeSpec=constraint.ConstraintsIntersection(constraint.SingleValueConstraint(0,3),
                      constraint.SingleValueConstraint(6,8)))
>>> i1.isSameTypeWith(i2)
False
>>> i1.isSuperTypeOf(i2)
True
>>>

While complete documentation on the API to all these ASN.1 items is not yet written, please, refer to example uses and source code for additional information.

Codec notes

In ASN.1 context, codec is a program that transforms between concrete data structures and a stream of octets suitable for transmission over the wire. This serialized form of data is sometimes called substrate or essence.

One of the properties of a codec is its ability to cope with incomplete data and/or substrate what implies codec to be stateful. In other words, when decoder runs out of substrate and data item being recovered is still incomplete, stateful codec would suspend and complete data item recovery whenever the rest of substrate becomes available. Similarly, stateful encoder would encode data items in multiple steps waiting for source data to arrive.

Codec restartability is especially important when application deals with large volumes of data and/or runs on low RAM.

For an interesting discussion on codecs options and design choices, refer to Apache ASN.1 project .

As of this writing, codecs implemented in pyasn1 are all stateless, mostly to keep the code simple.

The pyasn1 package currently supports BER codec and its derivates -- CER and DER. Encoder is used for transforming ASN.1 object into substrate:

>>> from pyasn1.type import univ
>>> from pyasn1.codec.ber import encoder
>>> encoder.encode(univ.Integer(12))
'\x02\x01\x0c'
>>>

while decoder recovers ASN.1 objects from substrate:

>>> from pyasn1.codec.ber import decoder
>>> decoder.decode('\x02\x01\x0c')
Integer(12), ''
>>>

Depending of encoding and tagging methods used, decoder may require to know ASN.1 syntax of data structure to be decoded. For example, PER-encoded or implicitly tagged values would not recover from substrate without knowing ASN.1 syntax of encoded data. Whenever decoder is given ASN.1 specification, this operation mode will be referred to as guided throughout this document.

The ASN.1 specification passed to decoder running in guided mode is simply a reference to the top-most ASN.1 type object of the concrete specification. Decoder would neither modify this specification object in any way nor use its current values, but rather use it as a pattern when creating new objects:

>>> from pyasn1.codec.ber import decoder
>>> decoder.decode('\x02\x01\x0c', asn1Spec=univ.Integer())
Integer(12), ''
>>>

One of the properties of BER codec is its use of either definite or indefinite length specification for serialized data. Although indefinite length form is especially important for stateful codec (which could produce&consume substrate in chunks), pyasn1 codecs fully support both length forms.

Constructed encoding is another feature of BER, closely related to indefinite length form. In essence, large scalar value (such as ASN.1 character or BitString type) could be chopped into smaller chunks by encoder and transmitted incrementally.

The following code would BER encode ASN.1 OctetString object using constructed (chopped by 4th octet) and indefinite length form:

>>> from pyasn1.type import univ
>>> from pyasn1.codec.ber import encoder
>>> encoder.encode(univ.OctetString('Quick brown fox'), defMode=0, maxChunkSize=4)
'$\x80\x04\x04Quic\x04\x04k br\x04\x04own \x04\x03fox\x00\x00'
>>>

Nothing special is required on decoding side to recover from various encoding forms. BER decoder transparently handles all of them.

Availability

The pyasn1 package is distributed under terms and conditions of BSD-style license. See LICENSE file in the distribution. Source code is freely available from project home.

Feedback

Comments and fixes are welcome at ilya@glas.net .