Anto Subash.

dotnet
abp
microservice
netcore6
.NET Microservice with ABP -Part : 6

DbMigration

This is sixth post of the series: .NET Microservice with ABP

Posts in the Series

Part 1. Initial Setup

Part 2. Shared Project

Part 3. Administration Service

Part 4. Identity Service

Part 5. SaaS Service

Part 6. DB Migration (this post)

Part 7. Yarp and Tye

Part 8. Identity server and Angular App

Part 9. Distributed event bus

Part 10. Docker and CI/CD

Part 11. Add a New service

Part 12. Central Logging

Table of contents

Add the references

For migrations we need to first add contracts and EF core projects as a reference in the db migrator. We need to do this for all the services.

1    <ItemGroup>
2        <ProjectReference Include="..\..\services\AdministrationService\src\Tasky.AdministrationService.Application.Contracts\Tasky.AdministrationService.Application.Contracts.csproj"/>
3        <ProjectReference Include="..\..\services\AdministrationService\src\Tasky.AdministrationService.EntityFrameworkCore\Tasky.AdministrationService.EntityFrameworkCore.csproj"/>
4        <ProjectReference Include="..\..\services\identity\src\Tasky.IdentityService.Application.Contracts\Tasky.IdentityService.Application.Contracts.csproj"/>
5        <ProjectReference Include="..\..\services\identity\src\Tasky.IdentityService.EntityFrameworkCore\Tasky.IdentityService.EntityFrameworkCore.csproj"/>
6        <ProjectReference Include="..\..\services\SaaSService\src\Tasky.SaaSService.Application.Contracts\Tasky.SaaSService.Application.Contracts.csproj"/>
7        <ProjectReference Include="..\..\services\SaaSService\src\Tasky.SaaSService.EntityFrameworkCore\Tasky.SaaSService.EntityFrameworkCore.csproj"/>
8    </ItemGroup>

We are adding EntityFrameworkCore and Contracts projects to the DbMigrations project.

Create DbMigrationService

Db migration service is where we can manage the migration and seeding of data of all the services in a single location.

1using System;
2using System.Collections.Generic;
3using System.Threading;
4using System.Threading.Tasks;
5using Microsoft.EntityFrameworkCore;
6using Microsoft.Extensions.DependencyInjection;
7using Microsoft.Extensions.Logging;
8using Tasky.AdministrationService.EntityFrameworkCore;
9using Tasky.IdentityService.EntityFrameworkCore;
10using Tasky.SaaSService.EntityFrameworkCore;
11using Volo.Abp.Data;
12using Volo.Abp.DependencyInjection;
13using Volo.Abp.EntityFrameworkCore;
14using Volo.Abp.Identity;
15using Volo.Abp.MultiTenancy;
16using Volo.Abp.TenantManagement;
17using Volo.Abp.Uow;
18
19namespace Tasky.DbMigrator;
20
21public class TaskyDbMigrationService : ITransientDependency
22{
23    private readonly ICurrentTenant _currentTenant;
24    private readonly IDataSeeder _dataSeeder;
25    private readonly ILogger<TaskyDbMigrationService> _logger;
26    private readonly ITenantRepository _tenantRepository;
27    private readonly IUnitOfWorkManager _unitOfWorkManager;
28
29    public TaskyDbMigrationService(
30        ILogger<TaskyDbMigrationService> logger,
31        ITenantRepository tenantRepository,
32        IDataSeeder dataSeeder,
33        ICurrentTenant currentTenant,
34        IUnitOfWorkManager unitOfWorkManager)
35    {
36        _logger = logger;
37        _tenantRepository = tenantRepository;
38        _dataSeeder = dataSeeder;
39        _currentTenant = currentTenant;
40        _unitOfWorkManager = unitOfWorkManager;
41    }
42
43    public async Task MigrateAsync(CancellationToken cancellationToken)
44    {
45        await MigrateHostAsync(cancellationToken);
46        await MigrateTenantsAsync(cancellationToken);
47        _logger.LogInformation("Migration completed!");
48    }
49
50    private async Task MigrateHostAsync(CancellationToken cancellationToken)
51    {
52        _logger.LogInformation("Migrating Host side...");
53        await MigrateAllDatabasesAsync(null, cancellationToken);
54        await SeedDataAsync();
55    }
56
57    private async Task MigrateTenantsAsync(CancellationToken cancellationToken)
58    {
59        _logger.LogInformation("Migrating tenants...");
60
61        var tenants =
62            await _tenantRepository.GetListAsync(includeDetails: true, cancellationToken: cancellationToken);
63        var migratedDatabaseSchemas = new HashSet<string>();
64        foreach (var tenant in tenants)
65        {
66            using (_currentTenant.Change(tenant.Id))
67            {
68                // Database schema migration
69                var connectionString = tenant.FindDefaultConnectionString();
70                if (!connectionString.IsNullOrWhiteSpace() && //tenant has a separate database
71                    !migratedDatabaseSchemas.Contains(connectionString)) //the database was not migrated yet
72                {
73                    _logger.LogInformation($"Migrating tenant database: {tenant.Name} ({tenant.Id})");
74                    await MigrateAllDatabasesAsync(tenant.Id, cancellationToken);
75                    migratedDatabaseSchemas.AddIfNotContains(connectionString);
76                }
77
78                //Seed data
79                _logger.LogInformation($"Seeding tenant data: {tenant.Name} ({tenant.Id})");
80                await SeedDataAsync();
81            }
82        }
83    }
84
85    private async Task MigrateAllDatabasesAsync(
86        Guid? tenantId,
87        CancellationToken cancellationToken)
88    {
89        using (var uow = _unitOfWorkManager.Begin(true))
90        {
91            if (tenantId == null)
92            {
93                /* SaaSService schema should only be available in the host side */
94                await MigrateDatabaseAsync<SaaSServiceDbContext>(cancellationToken);
95            }
96
97            await MigrateDatabaseAsync<AdministrationServiceDbContext>(cancellationToken);
98            await MigrateDatabaseAsync<IdentityServiceDbContext>(cancellationToken);
99
100            await uow.CompleteAsync(cancellationToken);
101        }
102
103        _logger.LogInformation(
104            $"All databases have been successfully migrated ({(tenantId.HasValue ? $"tenantId: {tenantId}" : "HOST")}).");
105    }
106
107    private async Task MigrateDatabaseAsync<TDbContext>(
108        CancellationToken cancellationToken)
109        where TDbContext : DbContext, IEfCoreDbContext
110    {
111        _logger.LogInformation($"Migrating {typeof(TDbContext).Name.RemovePostFix("DbContext")} database...");
112
113        var dbContext = await _unitOfWorkManager.Current.ServiceProvider
114            .GetRequiredService<IDbContextProvider<TDbContext>>()
115            .GetDbContextAsync();
116
117        await dbContext
118            .Database
119            .MigrateAsync(cancellationToken);
120    }
121
122    private async Task SeedDataAsync()
123    {
124        await _dataSeeder.SeedAsync(
125            new DataSeedContext(_currentTenant.Id)
126                .WithProperty(IdentityDataSeedContributor.AdminEmailPropertyName, "admin@abp.io")
127                .WithProperty(IdentityDataSeedContributor.AdminPasswordPropertyName, "1q2w3E*")
128        );
129    }
130}

Update appsettings.json

The appsettings has a different structure form the normal structure you will usually see in the abp projects. We use this as a data source to seed the ApiScope, ApiResource and Identity server clients.

1{
2  "ConnectionStrings": {
3    "SaaSService": "User ID=postgres;Password=postgres;Host=localhost;Port=5432;Database=TaskySaaSService;Pooling=false;",
4    "IdentityService": "User ID=postgres;Password=postgres;Host=localhost;Port=5432;Database=TaskyIdentityService;Pooling=false;",
5    "AdministrationService": "User ID=postgres;Password=postgres;Host=localhost;Port=5432;Database=TaskyAdministrationService;Pooling=false;"
6  },
7  "ApiScope": [
8    "AuthServer",
9    "SaaSService",
10    "IdentityService",
11    "AdministrationService"
12  ],
13  "ApiResource": [
14    "AuthServer",
15    "SaaSService",
16    "IdentityService",
17    "AdministrationService"
18  ],
19  "Clients": [
20    {
21      "ClientId": "Tasky_Web",
22      "ClientSecret": "1q2w3e*",
23      "RootUrls": [
24        "https://localhost:7004"
25      ],
26      "Scopes": [
27        "SaaSService",
28        "IdentityService",
29        "AdministrationService"
30      ],
31      "GrantTypes": [
32        "hybrid"
33      ],
34      "RedirectUris": [
35        "https://localhost:7004/signin-oidc"
36      ],
37      "PostLogoutRedirectUris": [
38        "https://localhost:7004/signout-callback-oidc"
39      ],
40      "AllowedCorsOrigins": [
41        "https://localhost:7004"
42      ]
43    },
44    {
45      "ClientId": "Tasky_App",
46      "ClientSecret": "1q2w3e*",
47      "RootUrls": [
48        "http://localhost:4200"
49      ],
50      "Scopes": [
51        "AuthServer",
52        "SaaSService",
53        "IdentityService",
54        "AdministrationService"
55      ],
56      "GrantTypes": [
57        "authorization_code"
58      ],
59      "RedirectUris": [
60        "http://localhost:4200"
61      ],
62      "PostLogoutRedirectUris": [
63        "http://localhost:4200"
64      ],
65      "AllowedCorsOrigins": [
66        "http://localhost:4200"
67      ]
68    },
69    {
70      "ClientId": "AdministrationService_Swagger",
71      "ClientSecret": "1q2w3e*",
72      "RootUrls": [
73        "https://localhost:7001"
74      ],
75      "Scopes": [
76        "SaaSService",
77        "IdentityService",
78        "AdministrationService"
79      ],
80      "GrantTypes": [
81        "authorization_code"
82      ],
83      "RedirectUris": [
84        "https://localhost:7001/swagger/oauth2-redirect.html"
85      ],
86      "PostLogoutRedirectUris": [
87        "https://localhost:7001/signout-callback-oidc"
88      ],
89      "AllowedCorsOrigins": [
90        "https://localhost:7001"
91      ]
92    },
93    {
94      "ClientId": "IdentityService_Swagger",
95      "ClientSecret": "1q2w3e*",
96      "RootUrls": [
97        "https://localhost:7002"
98      ],
99      "Scopes": [
100        "SaaSService",
101        "IdentityService",
102        "AdministrationService"
103      ],
104      "GrantTypes": [
105        "authorization_code"
106      ],
107      "RedirectUris": [
108        "https://localhost:7002/swagger/oauth2-redirect.html"
109      ],
110      "PostLogoutRedirectUris": [
111        "https://localhost:7002"
112      ],
113      "AllowedCorsOrigins": [
114        "https://localhost:7002"
115      ]
116    },
117    {
118      "ClientId": "SaaSService_Swagger",
119      "ClientSecret": "1q2w3e*",
120      "RootUrls": [
121        "https://localhost:7003"
122      ],
123      "Scopes": [
124        "SaaSService",
125        "IdentityService",
126        "AdministrationService"
127      ],
128      "GrantTypes": [
129        "authorization_code"
130      ],
131      "RedirectUris": [
132        "https://localhost:7003/swagger/oauth2-redirect.html"
133      ],
134      "PostLogoutRedirectUris": [
135        "https://localhost:7003"
136      ],
137      "AllowedCorsOrigins": [
138        "https://localhost:7003"
139      ]
140    }
141  ]
142}

Create IdentityServerDataSeedContributor for identity server

1using System.Threading.Tasks;
2using Volo.Abp.Data;
3using Volo.Abp.DependencyInjection;
4
5namespace Tasky.DbMigrator;
6
7public class IdentityServerDataSeedContributor : IDataSeedContributor, ITransientDependency
8{
9    private readonly IdentityServerDataSeeder _identityServerDataSeeder;
10
11    public IdentityServerDataSeedContributor(IdentityServerDataSeeder identityServerDataSeeder)
12    {
13        _identityServerDataSeeder = identityServerDataSeeder;
14    }
15
16
17    public async Task SeedAsync(DataSeedContext context)
18    {
19        await _identityServerDataSeeder.SeedAsync();
20    }
21}

Create IdentityServerDataSeeder for reading json and create resource

This service seeds the ApiScope, ApiResource and Identity server clients for the Identity server.

ServiceClient is a class to parse the Identity server clients

1public class ServiceClient
2{
3    public string ClientId { get; set; }
4    public string ClientSecret { get; set; }
5    public string[] RootUrls { get; set; }
6    public string[] Scopes { get; set; }
7    public string[] GrantTypes { get; set; }
8    public string[] RedirectUris { get; set; }
9    public string[] PostLogoutRedirectUris { get; set; }
10    public string[] AllowedCorsOrigins { get; set; }
11}
1using System;
2using System.Collections.Generic;
3using System.Linq;
4using System.Threading.Tasks;
5using IdentityServer4.Models;
6using Microsoft.Extensions.Configuration;
7using Volo.Abp.Authorization.Permissions;
8using Volo.Abp.DependencyInjection;
9using Volo.Abp.Guids;
10using Volo.Abp.IdentityServer.ApiResources;
11using Volo.Abp.IdentityServer.ApiScopes;
12using Volo.Abp.IdentityServer.Clients;
13using Volo.Abp.IdentityServer.IdentityResources;
14using Volo.Abp.MultiTenancy;
15using Volo.Abp.PermissionManagement;
16using Volo.Abp.Uow;
17using ApiResource = Volo.Abp.IdentityServer.ApiResources.ApiResource;
18using ApiScope = Volo.Abp.IdentityServer.ApiScopes.ApiScope;
19using Client = Volo.Abp.IdentityServer.Clients.Client;
20
21namespace Tasky.DbMigrator;
22
23public class IdentityServerDataSeeder : ITransientDependency
24{
25    private readonly IApiResourceRepository _apiResourceRepository;
26    private readonly IApiScopeRepository _apiScopeRepository;
27    private readonly IClientRepository _clientRepository;
28    private readonly IConfiguration _configuration;
29    private readonly ICurrentTenant _currentTenant;
30    private readonly IGuidGenerator _guidGenerator;
31    private readonly IIdentityResourceDataSeeder _identityResourceDataSeeder;
32    private readonly IPermissionDataSeeder _permissionDataSeeder;
33
34    public IdentityServerDataSeeder(
35        IClientRepository clientRepository,
36        IApiResourceRepository apiResourceRepository,
37        IApiScopeRepository apiScopeRepository,
38        IIdentityResourceDataSeeder identityResourceDataSeeder,
39        IGuidGenerator guidGenerator,
40        IPermissionDataSeeder permissionDataSeeder,
41        IConfiguration configuration,
42        ICurrentTenant currentTenant)
43    {
44        _clientRepository = clientRepository;
45        _apiResourceRepository = apiResourceRepository;
46        _apiScopeRepository = apiScopeRepository;
47        _identityResourceDataSeeder = identityResourceDataSeeder;
48        _guidGenerator = guidGenerator;
49        _permissionDataSeeder = permissionDataSeeder;
50        _configuration = configuration;
51        _currentTenant = currentTenant;
52    }
53
54    [UnitOfWork]
55    public async virtual Task SeedAsync()
56    {
57        using (_currentTenant.Change(null))
58        {
59            await _identityResourceDataSeeder.CreateStandardResourcesAsync();
60            await CreateApiResourcesAsync();
61            await CreateApiScopesAsync();
62            await CreateClientsAsync();
63        }
64    }
65
66    private async Task CreateClientsAsync()
67    {
68        var clients = _configuration.GetSection("Clients").Get<List<ServiceClient>>();
69        var commonScopes = new[] {
70            "email",
71            "openid",
72            "profile",
73            "role",
74            "phone",
75            "address"
76        };
77
78        foreach (var client in clients)
79        {
80            await CreateClientAsync(
81                client.ClientId,
82                commonScopes.Union(client.Scopes),
83                client.GrantTypes,
84                client.ClientSecret.Sha256(),
85                requireClientSecret: false,
86                redirectUris: client.RedirectUris,
87                postLogoutRedirectUris: client.PostLogoutRedirectUris,
88                corsOrigins: client.AllowedCorsOrigins
89            );
90        }
91    }
92
93
94    private async Task CreateApiResourcesAsync()
95    {
96        var commonApiUserClaims = new[] {
97            "email",
98            "email_verified",
99            "name",
100            "phone_number",
101            "phone_number_verified",
102            "role"
103        };
104
105        var apiResources = _configuration.GetSection("ApiResource").Get<string[]>();
106
107        foreach (var item in apiResources)
108        {
109            await CreateApiResourceAsync(item, commonApiUserClaims);
110        }
111    }
112
113    private async Task CreateApiScopesAsync()
114    {
115        var apiScopes = _configuration.GetSection("ApiScope").Get<string[]>();
116        foreach (var item in apiScopes)
117        {
118            await CreateApiScopeAsync(item);
119        }
120    }
121
122    private async Task<ApiResource> CreateApiResourceAsync(string name, IEnumerable<string> claims)
123    {
124        var apiResource = await _apiResourceRepository.FindByNameAsync(name);
125        if (apiResource == null)
126        {
127            apiResource = await _apiResourceRepository.InsertAsync(
128                new ApiResource(
129                    _guidGenerator.Create(),
130                    name,
131                    name + " API"
132                ),
133                true
134            );
135        }
136
137        foreach (var claim in claims)
138        {
139            if (apiResource.FindClaim(claim) == null)
140            {
141                apiResource.AddUserClaim(claim);
142            }
143        }
144
145        return await _apiResourceRepository.UpdateAsync(apiResource);
146    }
147
148    private async Task<ApiScope> CreateApiScopeAsync(string name)
149    {
150        var apiScope = await _apiScopeRepository.FindByNameAsync(name);
151        if (apiScope == null)
152        {
153            apiScope = await _apiScopeRepository.InsertAsync(
154                new ApiScope(
155                    _guidGenerator.Create(),
156                    name,
157                    name + " API"
158                ),
159                true
160            );
161        }
162
163        return apiScope;
164    }
165
166    private async Task<Client> CreateClientAsync(
167        string name,
168        IEnumerable<string> scopes,
169        IEnumerable<string> grantTypes,
170        string secret = null,
171        IEnumerable<string> redirectUris = null,
172        IEnumerable<string> postLogoutRedirectUris = null,
173        string frontChannelLogoutUri = null,
174        bool requireClientSecret = true,
175        bool requirePkce = false,
176        IEnumerable<string> permissions = null,
177        IEnumerable<string> corsOrigins = null)
178    {
179        var client = await _clientRepository.FindByClientIdAsync(name);
180        if (client == null)
181        {
182            client = await _clientRepository.InsertAsync(
183                new Client(
184                    _guidGenerator.Create(),
185                    name
186                ) {
187                    ClientName = name,
188                    ProtocolType = "oidc",
189                    Description = name,
190                    AlwaysIncludeUserClaimsInIdToken = true,
191                    AllowOfflineAccess = true,
192                    AbsoluteRefreshTokenLifetime = 31536000, //365 days
193                    AccessTokenLifetime = 31536000, //365 days
194                    AuthorizationCodeLifetime = 300,
195                    IdentityTokenLifetime = 300,
196                    RequireConsent = false,
197                    FrontChannelLogoutUri = frontChannelLogoutUri,
198                    RequireClientSecret = requireClientSecret,
199                    RequirePkce = requirePkce
200                },
201                true
202            );
203        }
204
205        foreach (var scope in scopes)
206        {
207            if (client.FindScope(scope) == null)
208            {
209                client.AddScope(scope);
210            }
211        }
212
213        foreach (var grantType in grantTypes)
214        {
215            if (client.FindGrantType(grantType) == null)
216            {
217                client.AddGrantType(grantType);
218            }
219        }
220
221        if (!secret.IsNullOrEmpty())
222        {
223            if (client.FindSecret(secret) == null)
224            {
225                client.AddSecret(secret);
226            }
227        }
228
229        foreach (var redirectUrl in redirectUris)
230        {
231            if (client.FindRedirectUri(redirectUrl) == null)
232            {
233                client.AddRedirectUri(redirectUrl);
234            }
235        }
236
237        foreach (var postLogoutRedirectUri in postLogoutRedirectUris)
238        {
239            if (client.FindPostLogoutRedirectUri(postLogoutRedirectUri) == null)
240            {
241                client.AddPostLogoutRedirectUri(postLogoutRedirectUri);
242            }
243        }
244
245        if (permissions != null)
246        {
247            await _permissionDataSeeder.SeedAsync(
248                ClientPermissionValueProvider.ProviderName,
249                name,
250                permissions
251            );
252        }
253
254        if (corsOrigins != null)
255        {
256            foreach (var origin in corsOrigins)
257            {
258                if (!origin.IsNullOrWhiteSpace() && client.FindCorsOrigin(origin) == null)
259                {
260                    client.AddCorsOrigin(origin);
261                }
262            }
263        }
264
265        return await _clientRepository.UpdateAsync(client);
266    }
267}

Update the TaskyDbMigratorModule

1typeof(AdministrationServiceEntityFrameworkCoreModule),
2typeof(AdministrationServiceApplicationContractsModule),
3typeof(IdentityServiceEntityFrameworkCoreModule),
4typeof(IdentityServiceApplicationContractsModule),
5typeof(SaaSServiceEntityFrameworkCoreModule),
6typeof(SaaSServiceApplicationContractsModule)

Repo: https://github.com/antosubash/AbpMicroservice

Buy Me a Coffee at ko-fi.com