A matrix group is a group of invertable square matrices (see chapter Matrices). In GAP you can define matrix groups of matrices over each of the fields that GAP supports, i.e., the rationals, cyclotomic extensions of the rationals, and finite fields (see chapters Rationals, Cyclotomics, and Finite Fields).
You define a matrix group in GAP by calling Group (see Group)
passing the generating matrices as arguments.
gap> m1 := [ [ Z(3)^0, Z(3)^0, Z(3) ],
> [ Z(3), 0*Z(3), Z(3) ],
> [ 0*Z(3), Z(3), 0*Z(3) ] ];;
gap> m2 := [ [ Z(3), Z(3), Z(3)^0 ],
> [ Z(3), 0*Z(3), Z(3) ],
> [ Z(3)^0, 0*Z(3), Z(3) ] ];;
gap> m := Group( m1, m2 );
Group( [ [ Z(3)^0, Z(3)^0, Z(3) ], [ Z(3), 0*Z(3), Z(3) ],
[ 0*Z(3), Z(3), 0*Z(3) ] ],
[ [ Z(3), Z(3), Z(3)^0 ], [ Z(3), 0*Z(3), Z(3) ],
[ Z(3)^0, 0*Z(3), Z(3) ] ] )
However, currently GAP can only compute with finite matrix groups. Also computations with large matrix groups are not done very efficiently. We hope to improve this situation in the future, but currently you should be careful not to try too large matrix groups.
Because matrix groups are just a special case of domains all the set
theoretic functions such as Size and Intersection are applicable to
Set Functions for Matrix Groups).
Also matrix groups are of course groups, so all the group functions such
as Centralizer and DerivedSeries are applicable to matrix groups (see
chapter Groups and Group Functions for Matrix Groups).
GAP 3.4.4