The limit is controlled by the MaxValRange limits.
Migrating the group into the Identity Vault namespace will temporarily sync up the member lists but any subsequent modification of the group in Active Directory will cause the group to again be truncated to 5000 members in the Identity Vault.
This issue occurs due to a limitation in Microsoft's DirSync API. Microsoft Active Directory limits the number of values returned in response to DirSync LDAP queries to 5000 values. This is an Microsoft Active Directory hard limit and is not dependent on the MaxValRange parameter of the Domain Controller's LDAP policy in Active Directory (see Ntdsutil.exe)
The Active Directory DirXML Driver uses Microsoft Active Directory Directory Synchronization Control to poll Microsoft Active Directory for changes. When any change is detected on the group all changed attribute values - up to 5000 values - are returned.
For Active Directory whose AD Forest and domain are operating at or after "Windows Server 2003" Domain functional levels, implementation of the DIRSYNC_LDAP_INCREMENTAL_VALUES flag to the Microsoft Active Directory Directory Synchronization Control resolves this issue. This control was implemented on DirXML 3.5 AD Driver Patch 1 - 20070601, now replaced by the IDM 3.5.1 or later downloads.